Duration: 9 hours and 36 minutes

Affected Components: ZNY Wiki

Mar 26, 04:40:00 GMT+0
Investigating - The NYARTCC Wiki is unavailable at wiki.nyartcc.org We are currently investigating this incident..

Mar 26, 12:20:28 GMT+0
Identified - The wiki remains unavailable at this time, due to excessive load. Load shedding operations are in progress, and we will attempt to restore access as soon as possible. .

Mar 26, 13:00:39 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. Wiki services have become available at this time. .

Mar 26, 14:16:15 GMT+0
Resolved - This incident has been resolved..

Duration: 8 hours and 56 minutes

Affected Components: ZNY Wiki

Feb 9, 03:35:42 GMT+0
Resolved - This incident has been resolved..

Feb 8, 18:40:00 GMT+0
Investigating - The NYARTCC Wiki is currently unavailable. We are currently investigating this incident. Thank you for your patience..

Feb 8, 19:08:44 GMT+0
Identified - The NYARTCC Wiki has been experiencing unusally high load within the last 18 hours, and has currently exhausted it's available resources. We are continuing to work on a fix for this incident. Updates will be provided within 1h..

Feb 9, 02:37:56 GMT+0
Monitoring - The NYARTCC Wiki is recovering at this time, utilization has returned to levels within normal tolerances. We're monitoring the system for any further impacts for the time being. .

Duration: 3 hours and 12 minutes

Affected Components: VATUSA API

Feb 1, 00:00:00 GMT+0
Investigating - Elevated error rates are being observed on API endpoints from VATUSA. We are investigating. .

Feb 1, 01:47:01 GMT+0
Identified - Beginning around 1900 EST, the VATUSA API endpoints began reporting elevated error rates in response to our standard requests for controller and user data. There appears to be unscheduled maintenance ongoing on the VATUSA systems. There has been no ETA for restoration provided by the VATUSA technical team at this time. Certain parts of our webpage and integrations (TeamSpeak, Discord Bot, etc.) may take longer to update to changes in ratings, certifications, and so on while this maintenance is ongoing. Website login resolutions may take longer to complete than normal. Please exercise patience if your operation takes longer than usual to perform on NYARTCC web services. If an update is provided by VATUSA, we will share it here..

Feb 1, 02:57:38 GMT+0
Monitoring - VATUSA report that the issue has been resolved. We are monitoring for any further errors at this time. .

Feb 1, 03:11:49 GMT+0
Resolved - This incident has been resolved..

Duration: 13 hours and 18 minutes

Affected Components: ZNY Wiki

Dec 20, 12:31:56 GMT+0
Resolved - The NYARTCC Wiki is available at this time. .

Dec 19, 23:13:43 GMT+0
Investigating - The NYARTCC Wiki is currently unavailable. We are currently investigating this incident and will provide an update as soon as possible. .

Duration: 1 day and 21 minutes

Affected Components: VATSIM Data Feed

Nov 11, 16:32:54 GMT+0
Resolved - The datafeed issue appears to have resolved at this time..

Nov 11, 16:09:57 GMT+0
Identified - We have identified an issue with VATSIM's Data Feed (jsonV3), and the service is currently entirely unavailable as of approximately 11/1451z. We here at NYARTCC do not have any further troubleshooting ability, but please be aware that certain Data Feed dependent services (Who's Online, active connections tracking, etc.) will be out of date or unavailable until the issue is resolved by the VATSIM Tech Team. .

Nov 11, 23:09:17 GMT+0
Identified - The issue with the Data Feed is now occurring again, and some services may be affected while the outage is ongoing. .

Nov 12, 16:31:11 GMT+0
Resolved - This incident has been resolved..

Duration: 21 minutes

Affected Components: Main Website

Oct 5, 19:43:42 GMT+0
Resolved - This incident has been resolved..

Oct 5, 19:23:06 GMT+0
Investigating - We are aware of issues on our web services, and are actively investigating..

Oct 5, 19:31:18 GMT+0
Identified - We have identified that one of our web server instances has become unresponsive. A new instance is being deployed at this time, and we expect services to resume within a few minutes. We thank everyone for their patience. If you experience an issue accessing services hosted by NYARTCC, please wait, and try your request again in a few moments time..

Oct 5, 19:36:08 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. The web services have been restored, we will continue to observe the status and make further updates as required. .

Duration: 19 hours and 44 minutes

Affected Components: Main Website

Jul 31, 19:08:07 GMT+0
Resolved - .

Jul 31, 00:21:45 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. One of the ZNY web server instances was experiencing excessive load on its CPU. The instance has been replaced and service appears to be recovering. Thank you for your patience. .

Jul 30, 23:23:42 GMT+0
Investigating - This is an automated report: The ZNY website appears to be down. We are currently investigating this incident and will update with more information as soon as possible. .

Duration: 4 days, 10 hours and 2 minutes

Affected Components: VATUSA API

Jul 16, 18:26:58 GMT+0
Identified - We have identified an ongoing intermittent issue with the VATUSA API, specifically endpoints that provide data exchange about controllers in the division. This is affecting multiple services on our web systems, including the updating of personal details, ratings, facility memberships, and so on. Some requests are completing successfully, but others are not. While we have no visibility into VATUSA's operations on this front, we hope that they will resolve the incident soon. In the meantime, if an operation you attempt does not work as expected, please wait a moment and retry your request. We'll provide updates as we receive them from VATUSA's technical team. Incident Start Date: 11/1346z Reported: 12/2046z Reported: 16/1220z.

Jul 18, 04:52:06 GMT+0
Identified - The intermittent failures on the VATUSA API are continuing at this time. While the issue has been acknowledged by VATUSA staff, there is currently no timeframe for restoration. Your patience is appreciated while the outage is ongoing, and as before, if you experience a failure or delay with any operation you attempt on NYARTCC's web services, please retry your request after a few moments. .

Jul 21, 04:29:16 GMT+0
Resolved - This incident has been resolved..

Duration: 54 minutes

Affected Components: VATSIM API, TeamSpeak Bot, VATSIM Data Feed, Main Website

May 23, 18:49:12 GMT+0
Investigating - VATSIMs core services are down, including authentication, data feeds and more which affects a lot of our services..

May 23, 19:42:48 GMT+0
Resolved - VATSIM services seem to be mostly back or are slowly recovering. .

Duration: 12 hours and 37 minutes

Affected Components: ZNY Wiki

Apr 4, 12:25:20 GMT+0
Investigating - We are currently investigating this incident. The ZNY Wiki is currently unavailable. We will provide updates as information is available. .

Apr 4, 13:00:47 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. However, we do expect there to be more issues - this is unfortunately, not a permanent fix..

Apr 4, 18:25:31 GMT+0
Identified - We are continuing to work on a fix for this incident. The Wiki will be temporarily unavailable while we work on an Authentication issue. More updates will be provided. .

Apr 5, 01:02:03 GMT+0
Resolved - This incident has been resolved..

Duration: 8 hours and 40 minutes

Affected Components: TeamSpeak Bot, TeamSpeak Server

Apr 2, 14:50:00 GMT+0
Identified - The Teamspeak bot is currently unavailable. We are aware of the issue, and are working on a solution. .

Apr 2, 23:29:47 GMT+0
Resolved - This incident has been resolved..

Duration: 17 hours and 9 minutes

Affected Components: ZNY Wiki

Feb 23, 20:09:47 GMT+0
Investigating - This is an automated report: The ZNY website appears to be down. We are currently investigating this incident and will update with more information as soon as possible. .

Feb 23, 21:19:50 GMT+0
Identified - We're tracking an outage affecting the ZNY Wiki (SOPs, LOAs, etc.). We'll provide more information as soon as possible. .

Feb 24, 13:18:48 GMT+0
Resolved - This incident has been resolved..

Duration: 1 day, 2 hours and 35 minutes

Affected Components: VATUSA Website, VATUSA API, VATUSA Academy

Feb 15, 12:47:49 GMT+0
Identified - The VATUSA API and Website are currently unavailable. This affects the certain services on our website, including but not limited to, the synchronization of ratings, the submission of training records, among others. Please try your operation again later if you encounter any issues using our systems. .

Feb 15, 18:43:13 GMT+0
Identified - The outage is ongoing. VATUSA reports that there is an issue with an upstream provider. There is no ETA for restoration available at this time. Please refer to the VATUSA discord for details. <https://discord.com/channels/699659614928502815/699660268543803493/1207732824287875103>.

Feb 16, 15:23:11 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 2 minutes

Affected Components: Discord Bot, TeamSpeak Bot, Main Website

Dec 23, 06:31:12 GMT+0
Identified - UPDATE: Maintenance is now in progress. We are planning to migrate the main NYARTCC website from one application server to another during the overnight (Eastern US Time) hours on the morning of Saturday 23 December. Most users will not experience an interruption during this migration, but sometimes unexpected things happen. If you experience a difficulty between 23/0600z and 23/1200z, please exercise patience and try your operation again after a few moments. .

Dec 23, 06:00:00 GMT+0
Identified - We are planning to migrate the main NYARTCC website from one application server to another during the overnight (Eastern US Time) hours on the morning of Saturday 23 December. Most users will not experience an interruption during this migration, but sometimes unexpected things happen. If you experience a difficulty between 23/0600z and 23/1200z, please exercise patience and try your operation again after a few moments. .

Dec 23, 07:01:35 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 hour and 51 minutes

Affected Components: TeamSpeak Server

Nov 28, 20:17:51 GMT+0
Resolved - This incident has been resolved..

Nov 28, 19:24:52 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Nov 28, 18:26:57 GMT+0
Investigating - We are currently investigating this incident. The NYARTCC TeamSpeak server has become unreachable. .

Duration: 4 hours and 49 minutes

Affected Components: VATSIM Data Feed

Nov 8, 13:30:00 GMT+0
Identified - The VATSIM datafeeed (which supplies the information for 'Whos Online', statistics gathering, hours/coverage tracking, and connection/session tracking) is experiencing intermittent failures beginning from around 08/1330z. Every few minutes the feed will supply bad information and fail to parse in most tools. This is unfortunately outside of our control, and is up to the VATSIM tech team to investigate and resolve. We'll provide an update if one becomes available. In the meantime, connection tracking, including displays on the Who's Online view, and in the TS server may be slow to update, and statistics may be delayed or inaccurate. .

Nov 8, 16:34:31 GMT+0
Identified - The VATUSA Tech Team has identified an issue with an upstream provider and is awaiting resolution. No ETA has been given at this time for resultion. Datafeed requests are currently failing between 50% - 70% of the time. .

Nov 8, 17:09:09 GMT+0
Monitoring - The VATSIM Tech Team reports that a fix has been implemented. The datafeed is being served well-formed, though until the upstream service fully recovers, datafeed elements may be updated less frequently than normal. .

Nov 8, 18:19:06 GMT+0
Resolved - The Datafeed is stabilized at this time and services are operating normally. .

Duration: 7 hours and 52 minutes

Affected Components: , TeamSpeak Bot, VATSIM →

Aug 22, 21:43:51 GMT+0
Resolved - VATSIM claims this incident has been resolved..

Aug 22, 13:51:46 GMT+0
Investigating - The VATSIM datafeeed (which supplies the information for 'Whos Online', statistics gathering, hours/coverage tracking, and connection/session tracking) is experiencing intermittent failures beginning from around 22/1300z. Every few minutes the feed will supply bad information and fail to parse in most tools. This is unfortunately outside of our control, and is up to the VATSIM tech team to investigate and resolve. We'll provide an update if one becomes available. In the meantime, connection tracking, including displays on the Who's Online view, and in the TS server may be slow to update, and statistics may be delayed or inaccurate. .

Aug 22, 18:35:09 GMT+0
Identified - The VATSIM tech team has identified the issue with their upstream provider, who is experiencing a network outage at some of their facilities. The issue is ongoing, updates will be provided as they become available. .

Duration: 2 hours and 7 minutes

Affected Components: Main Website

Aug 17, 12:00:33 GMT+0
Resolved - This incident has been resolved..

Aug 17, 09:53:45 GMT+0
Investigating - This is an automated report: The ZNY website appears to be down. We are currently investigating this incident and will update with more information as soon as possible. .

Duration: 10 hours and 45 minutes

Affected Components: Discord Bot, TeamSpeak Server, API, TeamSpeak Bot, ZNY Wiki, Main Website, , Website →

Aug 17, 01:15:00 GMT+0
Monitoring - Around 1-3AM GMT, users notices technical issues with the the ZNY website, wiki and Teamspeak servers with the website taking longer than normal to load, or being completely unavailable. The issue has been resolved, but we're continuing to monitor for further problems, and investigating what caused the servers to go down. .

Aug 17, 12:00:12 GMT+0
Resolved - The affected services have been stable, following the failover and recovery of the database service. .

Duration: 2 hours and 57 minutes

Affected Components: API, , Website →

Jul 21, 23:34:00 GMT+0
Monitoring - At around 23:34z (19:34 ET) the core NYARTCC database saw an abrupt increase in connection count and load. This temporarily saw some connection attempts refused, and as a result some sessions were interrupted. The database load has returned to normal levels at this time, and we'll continue to monitor. .

Jul 22, 02:31:13 GMT+0
Resolved - This incident has been resolved. Load on the database servers has returned to normal levels..

Duration: 13 hours and 24 minutes

Affected Components: , , Website → VATUSA →

Jul 6, 17:33:23 GMT+0
Monitoring - This incident is still open. The VATUSA tech team is investigating and has not provided an ETA for resolution at this time. .

Jul 6, 14:30:46 GMT+0
Monitoring - VATUSA is partially unavailable resulting in some functionality to our website being down such as submitting training notes, rating changes and more. .

Jul 6, 20:32:09 GMT+0
Monitoring - VATUSA is reporting that this incident has been resolved. We’ll continue to monitor for any further issues..

Jul 7, 03:55:09 GMT+0
Resolved - This incident has been resolved and VATUSA services are restored..