City of Dallas statement on network outage

While Court has been closed since May 3 due to the network outage, City of Dallas residents have been able to mail payments, requests, or documents to court.

All cases will be reset and notices will be mailed. Citation payments or documents due while Municipal Court is closed will be processed after May 30.

_________________________________________________________________________________________________________________

Public safety remains our highest priority. 911 has continually taken calls to dispatch Dallas Police Department (DPD) and Dallas Fire-Rescue (DFR) to protect and serve Dallas residents.

After public safety, public-facing services are being prioritized to minimize disruption of services to the public. There is still no indication that data from residents, vendors, or employees has been leaked.

911 calls are being entered into the Computer Assisted Dispatch (CAD) system. DPD has been dispatching automatically since Monday and DFR is routing calls for automatic dispatch where available. Progress continues while vehicles aren’t out on calls.

911 and 311 intake and dispatch continue via phone and radio dispatch while Computer Assisted Dispatch (CAD) components including 1900 mobile devices (1600 for DPD and 300 for DFR) and the server routing calls are tested to ensure no reinfection when redeployed. Completion of device cleaning to allow resumption of CAD is anticipated early this week

Please see below for frequently asked questions regarding the incident:

1. Will the City pay the ransom?

• The City is exploring all options to remediate this incident. As this is an ongoing criminal investigation, the City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.

2. Did the attackers send a ransom note to City printers?

• An image of information allegedly transmitted by an attacker has been published. While some media outlets removed or blurred a URL shown, others did not. Do not attempt to visit this URL as it may pose a threat to the device or network of anyone that does.

3. How did the breach initiate?

• This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker. The most common Ransomware attacks are initiated by exploiting vulnerabilities in software such as weak or default credentials, and social engineering (e.g. phishing) which tricks users into divulging confidential or personal information that may be used for fraudulent purposes.

4. How many devices are affected?

• This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.

5. What departments are affected and how?

• See https://www.dallascitynews.net/ for detailed updates.

6. What are next steps?

• The City’s IT Department continues to work with the assistance of cybersecurity experts and vendors to review software, servers, and devices to ensure they are uninfected before they are returned to service.

7. Are 9-1-1 dispatchers still having to take down information by hand and share it over a radio to respond?

• As prepared for and practiced in advance, 9-1-1 operators continue to answer and dispatch calls utilizing back up procedures and the City’s public safety radio system. As City staff and contractors review devices, ensure they are secure, and bring them online, Computer Assisted Dispatch (CAD) functionality will increase for DPD, DFR and 311.

8. What do people on deadline to pay citations need to do?

• Citation payments and documents due while the Municipal Court system is down will be accepted after service is restored. Anyone who has a citation to pay or documents due while Municipal Court is closed will be granted an extension to pay or present the documents to the Court without penalty.

9. How long this will take to get resolved?

• The City’s IT Department will continue to work with the assistance of cybersecurity professionals and our system vendors to review software, servers, and devices to ensure they are uninfected and can be returned to service as quickly as possible.

10. Is the prior network outage related to this attack?

• There is no evidence that the network outage occurring on April 19, 2023 is related to this ransomware attack. The network outage of April 19 was caused by a hardware failure. The City and its network services provider AT&T resolved that outage.

11. Has Royal made any demands from the city of Dallas? Has the city learned how Royal was able to gain access to its systems? If so, how did the group compromise those servers? If so, what is the group demanding, and what has been the city’s response?

• This is an ongoing criminal investigation. The City cannot comment on specific details which risk impeding the investigation or exposing vulnerabilities that can be exploited by an attacker.

12. Should residents be concerned that their personal information could be leaked online as a result of this attack? Why or why not?

• At this time the City has no indication that customer information such as billing data or personally identifiable information (PII) has been leaked from City systems or databases. Should this change, the City will notify potentially impacted individuals with information and instructions. Should any individual be contacted by someone claiming to represent the City of Dallas and ask for a payment or personal information, please take note of the number they are calling from and the number they are reaching, then hang up and call the City Department to report.

13. How much is this attack costing the City?

• As this is an ongoing criminal investigation, a determination of financial impact cannot be given at this time.

14. Were social media channels compromised as a result of the outage?

• The City of Dallas social media accounts have not been compromised. Updates will continue to be shared via DallasCityNews.net.

The City of Dallas’ preparation for security threats and early response to network outages caused by a cyberattack May 3 have maintained essential City services for residents throughout the week.

Since Wednesday morning, when a group called Royal initiated an illegal attack targeting the City, Dallas’ Information and Technology Services (ITS) department and ITS’ expert cybersecurity vendors have effectively worked nonstop to swiftly isolate a virus and gradually restore service, prioritizing public safety and resident-facing departments. Much progress has been made, but the recovery process is ongoing.

Thanks to heroic teamwork by our first responders, key public safety functions continue as usual despite technical difficulties. 911 and 311 calls are being answered and Dallas Police Department and Dallas Fire-Rescue are being timely dispatched by radio.

For residents with non-emergency needs, 311 is still taking service requests by phone but the OurDallas app and online portal are temporarily unavailable. Another option is service in person at City facilities during regular business hours.

Sanitation collection remains on schedule and disposal sites are operational during regular business hours. If service is missed, please call 311.

Dallas Water Utilities service is unaffected, and disconnections are discontinued until the outage is resolved. Statements may be paid by mail; however, for those who prefer to pay in-person or online, late fees will not be charged for payments that cannot be processed until service is restored. DWU autopay will draft when service is restored. If you have questions or need assistance, please call (214) 651-1441 or walk into the water lobby at City Hall Monday through Friday between 8 a.m. to 5 p.m.

Office of Community Care clinics for Women, Infants and Children (WIC) are open and providing benefits. Vital Statistics is issuing records, but to ensure any records sought are available before you arrive, please call 214-670-3092.

Dallas Public Library branches are open, operational, and can check out media to residents with a library card; however, residents with media due to be returned are asked to hang on to it a little longer. There will be no late fees charged for materials due during the service outage. Digital media is also available via Hoopla and Overdrive. Internet-connected computers may be limited, so users in need of online device access should call ahead to their respective branch.

Dallas Animal Services continues handling adoptions, fosters, rescues, and returns to owners in-person on a case-by-case basis at 1818 N. Westmoreland Road, 75212.

Code Compliance is issuing garage sale permits only in-person at their headquarters at 3112 Canton Street, 75226.

While pages on the City’s website are being restored, Special Events permit requests may be submitted through the following direct links:

• Special Event, Commercial Promoter, and Fair Park Parking License Application
• Street Pole Banner Application
• Commercial Filming
• Neighborhood Market

Development Services can review paper plans for walk-ins at 320 E. Jefferson Blvd., 75203 during regular business hours. However, while Permitting, Public Works, and Zoning application and payment systems are offline, submissions cannot be received or approved. This is a dynamic situation, and patience is appreciated while we focus on expediting full-service restoration.

Municipal Court remains closed Monday, May 8. There will be no court hearings and no trials. Cases scheduled during this outage will be reset, and updates will be mailed. Citation payments and documents due while Municipal Court is closed will be accepted after service is restored.Please note, no one from the City of Dallas will reach out to members of the public to ask for payment in person or by phone. Never give out your password or payment information by phone or through an email link. To protect against cyber threats please install the Dallas Secure app on your iOS or Android device. If you are contacted by someone seeking payment who claims to be from a City of Dallas department, please take note of the number they are calling from and the number they reached you on, then hang up and call the City of Dallas department they claim to be from to report this potential impersonation.

For years the City has monitored to ensure that residents’ and vendors’ information is not leaked. If it were, we would advise the public immediately. At this point, we do not believe that’s the case.  Please pardon the inconvenience, thank you again, and stay safe!

4:30 p.m. update, Thursday, May 4

Services to residents continue with some delays and disruption due to the outage. Departmental updates as of 4:30 p.m. are as follows:

ITS isolated the issue and is gradually restoring service prioritizing public safety and resident-facing departments.

DPD and DFR service continues as usual.

311 and 911 calls continue to be received and dispatched.

Dallas Water Utilities is unable to process payments. Disconnections will be discontinued until the outage is resolved.

Vital Statistics is issuing records with limited capacity. Some records may be unavailable, especially from prior to 2005. Residents may contact Vital Records at 214-670-3092.

Women, Infants & Children (WIC) is maintaining normal operations at all clinics and able to issue benefits.

Courts remain closed and cases will be reset; jurors do not need to report for service and notices will be sent by mail.

Code Compliance Services response to service requests may be delayed. Code is currently unable to process Single-Family and Multi-Tenant registrations. Garage sale permits can be issued in-person only at 3112 Canton St.

Dallas Animal Services is responding to injury and emergency requests and non-emergency response is delayed. DAS is handling adoptions, fosters, rescue and return to owners on a case-by-case basis at 1818 N. Westmoreland Rd. during regular business hours.

For Office of Special Events permits please use the following links to login and create application:

• Special Event, Commercial Promoter, and Fair Park Parking License Application
• Street Pole Banner Application
• Commercial Filming
• Neighborhood Market

City Secretary’s Office Open Records Requests will be delayed.

Development Services, Permitting, Public Works, and Zoning applications and payments cannot be received, and permits cannot be issued.

Update as of 10 a.m. May 4.
ITS and its vendors continue to work around the clock to contain the outage and restore service, prioritizing public safety and public-facing departments. A group called Royal initiated the attack. Chief Zielinski will brief the Public Safety Committee Monday, May 8.

As of 10 a.m. these are service updates:

• DPD and DFR service to residents is unaffected.
• 911 calls continue to be received and dispatched.
• 311 Calls are being answered but non-emergency service requests may be delayed.
• Courts are closed and LiveChat is inaccessible. All cases will be reset; jurors do not need to report for service and notices will be sent by mail.
• CSO – Saturday’s election is unaffected; Dallas County will share official information including results. Meeting notices are being posted and meetings may be viewed at dallascityhall.webex.com, dallascitynews.net/watch-live, and Spectrum channels 16 & 95 and AT&T U-verse at 99. Contracts may be delayed.
• Dallas Public Library All branches are open and in-person checkouts continue; online materials are currently unavailable.
• DWU Billing is unaffected; meter reading will be delayed. Only IVR can take credit card payments. Disconnections will be discontinued until the outage is resolved.

City Manager T.C. Broadnax has released the following statement:

“Since City of Dallas’ Information and Technology Services detected a cyber threat Wednesday morning, employees have been hard at work to contain the issue and ensure continued service to our residents. While the source of the outage is still under investigation, I am optimistic that the risk is contained. For those departments affected, emergency plans prepared and practiced in advance are paying off. We apologize for any inconvenience and thank residents for their understanding as we continue to work around the clock until this issue is addressed. For updates, please keep an eye on dallascitynews.net.”

Update as of 5 p.m. May 3.

ITS will continue working around the clock to identify the source of the outage and isolate impacted devices. Departmental service to residents continues with minimal disruption as follows.

• DPD & DFR: Service to residents is unaffected although the home page is offline.
• 911: Calls continue to be received and dispatched
• 311: Phone IVR is fully functional. 311 agents are accepting phone calls and dispatching Service Requests.
• Courts: Court was closed Wednesday and will also be closed on Thursday. All cases will be reset and notices will be sent by mail.
• DWU: Payment may be processed via IVR. Online payment processing may be delayed but no disconnections will be processed until the outage is resolved.

Currently less than 200 of the City’s thousands of devices are impacted, but if any City device is at risk, it will be quarantined and blocked by ITS. For compromised machines, restoration will prioritize public safety, anything public-facing, then all other departments.

DallasCityNews.net will be updated at least once a day until the incident is resolved.

Wednesday morning, the City’s security monitoring tools notified our Security Operations Center (SOC) that a likely ransomware attack had been launched within our environment.

Subsequently, the City has confirmed that a number of servers have been compromised with ransomware, impacting several functional areas, including the Dallas Police Department Website. The City team, along with its vendors, are actively working to isolate the ransomware to prevent its spread, to remove the ransomware from infected servers, and to restore any services currently impacted. The Mayor and City Council was notified of the incident pursuant to the City’s Incident Response Plan (IRP).

The City is currently working to assess the complete impact, but at this time, the impact on the delivery of City services to its residents is limited. Should a resident experience a problem with a particular City service, they should contact 311. For emergencies, they should contact 911.