[SERVERBUG] Internal error occurred (In Progress)
  • Priority - High
  • Affecting Other - Thexyz Webmail
  • UPDATE: We have made progress in repairing the impacted mailboxes and will continue to run the processes listed below:

    We auto detect the following issue by scanning system logs for errors that are generated when a user logs in to access their email:

    - for those users who have not logged in since the system load issue, there is a small chance that an error in the Inbox will cause them to receive an error upon login. This issue may also present itself with intermittent errors when attempting to connect to a mail client using POP or IMAP protocols. We are continuously scanning for these errors and automatically fixing them several times each day.

    Users may clean-up affected folders for the following issue:

    - if a user has extraneous folders beginning with the word "lost-", any email in these folders can be moved back to the original folder and the extraneous folder may be deleted.

    Administrators should contact the support team for the following issue:

    - if a user believes that there may be missing messages or folders, the administrator should contact support with that user’s email address & information regarding what appears to be missing. Because there are cases in which we cannot automatically detect this issue, contacting support is necessary.

    We do not anticipate issuing further updates for this incident.

    UPDATE: The primary focus early this week was to establish stable mail delivery and login access. While we have made significant progress, some work remains. Our current focus is on individual mailbox repairs due to issues indirectly caused when the system was under high load. Now that the system is stabilized, these mailbox repairs are processing more quickly. While we can automatically detect and remediate some issues, others require you to contact the support team.

    There are certain issues which we are unable to detect If a user has not logged in. Because some users may go days or even weeks without logging in, we anticipate the symptoms described below will persist over a period of time. Administrators should contact the support team for the following issue:

    - if a user believes that there may be missing messages or folders, the administrator should contact support with that user’s email address & information regarding what appears to be missing. Because there are cases in which we cannot automatically detect this issue, contacting support is necessary.

    We auto detect the following issues by scanning system logs for errors that are generated when a user logs in to access their email:

    - for those users who have not logged in since the system load issue last week, there is a small chance that an error in the Inbox will cause them to receive an error upon login. This issue may also present itself with intermittent errors when attempting to connect to a mail client using POP or IMAP protocols. We are continuously scanning for these errors and automatically fixing them several times each day.

    - if a user has extraneous folders beginning with the word "lost-" these errors are automatically detected by our log scan. While this requires no action on your part to be cleaned, the user may also safely delete them if they do not contain any email. If a folder containing email is deleted, users may check the Trash or Recover Deleted Items folders to retrieve the email.

    These processes will continue to run through the weekend. We expect to provide the next update on Monday, July 24, 2017.

    UPDATE: We continue to perform mailbox level cleanups to address the following issues, which we have identified proactively from system logs:

    - mailboxes with login/access issues: this issue may cause intermittent errors for users that connect via POP or IMAP.
    - extraneous folders prefixed with "lost-"
    - missing messages or folders: the information is present, but not visible to email programs, including Webmail.

    We recommend that those users who experience intermittent errors connecting via POP or IMAP utilize Webmail instead.

    While we are working to resolve all symptoms, our highest priority is ensuring all affected users have access to their mailbox.

    UPDATE: The initial passes to fix affected mailboxes were successful and the hosted email environment has stabilized.
     
    We will continue to actively run the bulk mailbox level cleanup to address the following issues we have identified proactively from system logs:
     
    -  missing messages or folders; the information is present but not visible to email programs including Webmail
    -  extraneous folders starting with "lost-"
    -  mailboxes that we detect as having login/access issues 
     
    Some mailboxes may require additional attention to fully resolve, and our team is actively working through those.
     
    We will provide another update to the mailbox cleanup efforts tomorrow afternoon.

    UPDATE: Our initial pass of fixes for mailboxes that we detected as having login/access issues was successful. We will run a pass for newly discovered mailboxes that our logs have indicated are having issues.

    Once complete, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:

    - missing messages or folders; the information is present but not visible to email programs including Webmail
    - extraneous folders starting with "lost-"

    As we continue to progress through our mailbox cleanup efforts, some users may see slowness within Webmail.

    Some mailboxes may require additional attention to resolve some of the issues experienced, and our team is actively working through those.

    We will provide updates as we progress through these cleanup efforts.

    UPDATE: We are continuing to focus on maintaining access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load.
     
    We have begun running a fix to address the issues for users that have not yet been able to successfully access their mailbox.  During this time, some users may see some slowness within Webmail.
     
    Once completed, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:
     
    - missing messages or folders; the information is present but not visible to email programs including Webmail
    - extraneous folders starting with “lost-“
     
    Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

    We will continue to work on these mailboxes through the evening.  We will provide our next update during the morning of Tuesday, July 18th.

    UPDATE: Over the weekend we have worked through a system-wide cleanup to address the after-effects of the high system load experienced last week. The load was the result of an unknown bug that impacted an effort to proactively double system capacity. Please see the system status update posted 06:49 PM EDT on 07/15/17 for a brief, preliminary review of the causes of the system disruption.

    Our priority today is to maintain good access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. In an abundance of caution, we are limiting the number of simultaneous connections to a single mailbox to 2 POP connections and 4 IMAP connections. Mail programs that exceed the limit will usually retry and users are unlikely to see errors due to connection limits.

    We are also rate limiting email delivery for mailboxes that receive hundreds or thousands of email messages per minute; that email will safely queue for delivery over a longer period of time. If a user normally receives less email than this, they should see normal delivery speeds.

    After peak load hours today, usually finished before 2pm Eastern, we will consider a bulk mailbox level cleanup to address several issues we have identified proactively from system logs. These issues include:

    - some users have not yet been able to successfully access their mailbox
    - missing messages or folders; the information is present but not visible to email programs including Webmail
    - extraneous folders starting with “lost-“

    Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

    We will continue to provide updates as they become available.

    UPDATE: Our efforts this weekend have been focused on system wide cleanup to address the consequences of high system load after expanding capacity. These efforts are nearing completion. We will soon begin a more detailed mailbox-level cleanup effort to address some issues that we have seen among some users. While those updates happen there may be very brief periods where accessing email is slow or gives an error; trying again a few minutes later should be successful.
     
    The mailbox level cleanup will address, in an automated fashion, the following issues for users we have been able to identify proactively:

    - some users have not yet been able to successfully access their mailbox
    - missing messages or folders; the information is present but not visible to email programs including Webmail
    - extraneous folders starting with “lost-“
     
    Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.
     
    We will continue to provide updates as they become available.

    UPDATE: Previous updates have been focused on resolution status as we worked through issues from the evening of Wednesday, July 12, 2017 through today, Saturday, July 15. We have compiled a preliminary view into the causes behind the system issues this week, and will have a final report available in the coming weeks. We recognize that these issues were significant for many of our customers, and we apologize for the difficulties you have experienced.

    UPDATE: System load is progressing towards normal levels and most users should find the email system operational. Until the system is fully restored, users who experience occasional timeouts and connection errors should refresh their connection.

    We have made progress in repairing the impacted mailboxes. Some users may experience sporadic delays today while the repair continues.

    UPDATE: Impacted users no longer need contact the support team to provide mailbox details. We continue to monitor the environment and will apply fixes to resolve the aforementioned symptoms for affected users.

    We will continue to provide updates as they become available.

    UPDATE: We are continuing to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders.
    - POP/IMAP access issues.

    There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017. While some of these symptoms may resolve themselves over time, if you are experiencing the below symptoms over a prolonged period of time , please contact Support for further assistance. Symptoms include but are not limited to:

    - Unread Items Issue: A large number (or entire folder) of messages has changed from "Read" to "Unread"
    - Missing Subfolders Issue: Missing Subfolders that seemingly disappeared without intervention. (These folders are hidden and not actually missing)
    - “Lost” Folders Issue: User will report a group of folders that have "lost" in the folder name followed by a string of characters "Ex: lost-asfkjl3q5kjafls"
    - POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined).
    - Folder Deletion Issue: Users unable to delete folders in Webmail. Error received will be "The selected folder does not exist...."
    - Webmail Access Issue: Users unable to access their email through webmail or Outlook and receive a hard hat or "an error occurred ..." message.
    - Message duplication issue: Users may experience duplication of recently sent messages.

    UPDATE: At this time, message delivery times and webmail responsiveness have remained stable but we will continue to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders.
    - POP/IMAP access issues.

    There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017. While some of these symptoms may resolve themselves over time, if you are experiencing the below symptoms over a prolonged period of time , please contact Support for further assistance. Symptoms include but are not limited to:

    - Unread Items Issue: A large number (or entire folder) of messages has changed from "Read" to "Unread"
    - Missing Subfolders Issue: Missing Subfolders that seemingly disappeared without intervention. (These folders are hidden and not actually missing)
    - “Lost” Folders Issue: User will report a group of folders that have "lost" in the folder name followed by a string of characters "Ex: lost-asfkjl3q5kjafls"
    - POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined).
    - Folder Deletion Issue: Users unable to delete folders in Webmail. Error received will be "The selected folder does not exist...."
    - Webmail Access Issue Users unable to access their email through webmail or Outlook and receive a hard hat or "an error occurred ..." message. 

    The next update will be provided by 10:00 AM EST / 2:00 PM GMT.
     
    UPDATE: Message delivery times remain at normal levels.

    System performance has stabilized to normal levels. We will continue to monitor the environment as system load increases throughout the day.

    As system load increases throughout the day users may experience:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders.
    - POP/IMAP access issues.

    The next update will be provided by 8:00 AM EST/ 12:00 PM GMT.
    Posted on 07/14/2017 3:42 AM EST
    Message delivery times remain at normal levels.

    UPDATE: We continue to improve system performance across our Hosted Email environment.

    During this increased system load, users may continue to see the following:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders.
    - POP/IMAP access issues.

    The next update will be provided by 5:00 AM EST / 9:00 AM GMT.
     
    UPDATE: Message delivery times have been restored to normal levels.

    We continue to monitor system load and actively improve system performance across our Hosted Email environment.

    During this increased system load, users may continue to see the following:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders.
    - POP/IMAP access issues.

    The next update will be provided by 3:00 AM EST / 7.00 AM GMT.
     
    UPDATE: We continue to monitor system load, and are actively improving system performance across our Hosted Email environment.

    During this increased system load, users may continue to see the following:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders.
    - Delivery delays continue to fall and are nearing resolution.
    - POP/IMAP access issues.

    These symptoms will resolve themselves once the issue has been resolved.

    UPDATE: We continue to monitor system load, and are actively improving performance across our Hosted Email Platform.

    During this increased system load, users may continue to see the following:
    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

    - Latency in accessing mail or mail folders.

    - Delivery delays (currently 10 minutes).

    - POP/IMAP access issues.

    These symptoms will resolve themselves once the issue has been resolved. Next update will be provided by 10:00 PM EST.

    UPDATE: No additional updates or ETA available at this time. Our teams continue to work on resolving the issue. Users may continue to experience the symptoms below.

    UPDATE: Our teams continue to actively work on reducing load and returning the system to normal operation. In the meantime, users affected by this may notice the following:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders
    - Delivery delays (approximately 37 minutes)
    - Intermittent access to POP/IMAP

    These specific symptoms will resolve themselves once the issue has been resolved. Another update will be posted by 4:00 PM EST.

    UPDATE: Our teams are actively working to reduce load and return to normal operation. In the meantime, users affected by this may also experience the following:

    - Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
    - Latency in accessing mail or mail folders
    - Delivery delays (approximately 22 minutes)

    These specific symptoms will resolve themselves once the issue has been resolved.

    Due to an emergency maintenance that was performed at 9:00 PM EST 7/12/2017, a portion of users may notice a disruption of service with their Hosted Email mailbox.

    Some of the symptoms that are being reported would include:

    - Latency when navigating the Webmail interface
    - Slowness upon logging in to the Webmail interface
    - Delayed email delivery
    - POP Error 'Your incoming (POP3) email server has reported an internal error. If you continue to receive this message, contact your server administrator or Internet service provider (ISP). The server responded:
    - ERR [SYS/TEMP] Internal error occurred. Refer to server log for more information.
    - Users unable to delete folders in Webmail "The selected folder does not exist...."
    - Messages previously marked as read are now unread
    - "Lost Folders" appear in webmail
    - Delayed email delivery

    While some of these symptoms may resolve themselves over time, if you need immediate assistance or if you are experiencing these symptoms over a prolonged period of time , please contact Support for further assistance.




  • Date - 13/07/2017 11:55
  • Last Updated - 25/07/2017 10:35
Emergency Maintenance - Hosted Email (Scheduled)
  • Priority - High
  • Affecting Other - Thexyz Webmail
  • As part of our efforts to continually improve our product offerings, we will be performing a maintenance on our Hosted Email platform. The maintenance window is scheduled to occur Wednesday, July 12th between 9:00 PM and 10:00 PM EDT.

    While all measures are being taken to prevent any impact, customers may experience a brief disruption of service during the maintenance window.

    We encourage administrators with users experiencing issues once the maintenance window concludes to contact Thexyz Support Team for further troubleshooting.

  • Date - 12/07/2017 21:00 - 12/07/2017 22:00
  • Last Updated - 12/07/2017 09:00
Maintenance - Control Panel (Scheduled)
  • Priority - Low
  • Affecting Other - Email Admin Portal
  • We will be performing a planned maintenance on our Control Panel.

    The maintenance window is scheduled to occur Monday, June 26th, 2017 from 8:00 PM EST to 10:00 PM EST.

    During this time, administrators may experience delays when creating or modifying Hosted Exchange mailboxes.

    We encourage administrators that experience these delays to wait until the maintenance has completed before attempting to resubmit any additional changes.

  • Date - 26/06/2017 20:00 - 26/06/2017 22:00
  • Last Updated - 24/06/2017 10:28