Today, from around 12:00 PM CDT, Google Cloud Platform reported a global incident affecting multiple regions and services. At WeShip we observed delays or partial failures in features such as order status updates, label generation, and metric queries. Thanks to our alternate providers and contingency measures, most of our services returned to normal within the following hours.
Key Details:
- Incident start (approx. 12:00 PM CDT): Google Cloud detected issues in many regions (including us-central1, various regions in the Americas, Europe, Asia, and Oceania). This caused downtime or slow responses in infrastructures and APIs hosted on GCP.
- Impact on WeShip (~20% of services):
- Order status updates: intermittent delays or errors.
- Label generation and automatic notifications: some delays or temporary failures.
- Tracking metric queries and API integrations: unstable behavior or high wait times.
- Progressive recovery:
- By 2:00 PM PDT (approx. 4:00 PM CDT), Google confirmed mitigations in us-central1 and several zones showed recovery signs.
- By 2:28 PM PDT (approx. 4:28 PM CDT), WeShip services were restored.
- Around 3:23 PM PDT (approx. 5:23 PM CDT), most regions had recovered; a few services had residual impact, resolved within an hour after that.
- Current status:
- Most WeShip features are now operating normally. If you experience unusual behavior, try refreshing or retrying later. Our monitoring indicates stability across nearly all systems.
- Actions taken by WeShip:
- Continuous monitoring and communication with Google Cloud support.
- Temporary rerouting of critical workloads to alternate providers when possible.
- Reinforced support team to handle individual customer inquiries.
- Recommendations for users:
- Check your order flows: if you see synchronization errors, retry after a few minutes or contact support.
- For non-urgent processes, consider postponing until full normalization is confirmed.
- Check our status dashboard (usual link) for real-time updates.
- Next steps:
- We will refine high-availability and accelerated recovery plans, leveraging our multiple infrastructure providers.
- Gratitude and commitment:
- We apologize for any inconvenience this outage may have caused. Our team is focused on maintaining your operations’ continuity and improving service resilience. We will keep you informed via email and the status dashboard as soon as there are updates or final fixes.
