Hello everyone! It's your friendly neighborhood red panda, back again with another Development Log, where we'll be diving into the nerdy technical stuff, sharing some behind-the-scenes peeks of the SkyBlock: End & Beyond update, and previewing what comes next.
Lux, our server software
Just like how we did in Dev Logs #2, I want to start by going over some of the work we've done on Lux since that article:- Implemented a brand new mob AI system, currently powering mobs in The End and soon expanding to the rest of the server, starting with some new mobs arriving in SkyBlock v0.5.1.
- Added a lot more new blocks, including the ones from recent Minecraft updates and custom ones.
- Added support for DDUI. DDUI is Minecraft's new UI system based on Ore UI, but it still lacks the customization options our existing UIs have. We'll wait for Mojang to add more customization before switching over completely.
- Migrated 9 more packets to CloudburstMC/Protocol. Only two more to go!
- Migrated Wool (our custom level format) worlds to store NBT states rather than arbitrarily mapped ID and meta values.
- Migrated entities to namespaced IDs instead of numeric ones.
- Migrated effects to namespaced IDs instead of numeric ones.
- Migrated enchantments to namespaced IDs instead of numeric ones.
- Improved multi-version support, made easier by using the existing CloudburstMC/BlockStateUpdater.
- Rewrote the effect system to be behavior-driven instead of inheritance-based.
- Added an entity, item, and block tag system, which lets us filter things by their grouping tags instead of doing manual object type checks.
- Added the spark profiler in place of the legacy Aikar's profiler.
- Fixed certain lighting update issues.
- Optimized arrow collisions.
- Improved the Block break and collision API.
- This one was a major headache. Nukkit's existing Block API treats collision as the block itself, which is a strange approach that limits how custom blocks can be implemented.
- The break API was similarly scattered, with legacy, confusing names for things like speed and drops. These changes aim to improve readability and maintainability across the codebase.
- Added a centralized query system for blocks and entities, as well as raycasting for block checks.
- The existing systems were confusing and couldn't be optimized any further for memory and CPU usage. The new API gives us headroom for both immediate improvements and future feature work.
- And more that we honestly lost track of!
The lost items incident
We need to talk about the lost items incident from a few months ago: what happened, why it happened, and what we've done to prevent it from happening again.To help you understand the issue, I first need to explain how CraftersMC runs. Here's a simplified graphic of the architecture:
Aside from the services shown as blue cards, almost everything depends on ZenControl, our central backend service. Gaining XP, saving your inventory, purchasing something with Gems, starting an account upgrade and countless other actions go through ZenControl.
This means ZenControl always has to be online, which is why we sometimes have to bring the server down for maintenance even for what seems like a tiny change. If something needs to change on ZenControl, everything has to stop.
It also means that if something happens to ZenControl, such as a crash, none of these actions can be processed.
The game servers don't write SkyBlock profile data directly to the database. Instead, ZenControl keeps active SkyBlock profiles cached in memory. Every change you make is applied to that in-memory version first. ZenControl then writes those changes to the database every few minutes, as well as whenever a profile has been inactive for long enough.
After the server infrastructure migration we've done in January 2026, we lowered ZenControl's maximum memory because it wasn't using anywhere close to the previous limit. Unfortunately, that turned out to be not the best decision.
With the lower memory limit, ZenControl would occasionally hit the limit and cause some threads to throw an
OutOfMemoryError. If this happened to the periodic save thread, profile saves would silently stop being written to the database, meaning the unsaved changes would simply be gone.That perfectly matched the reports we were receiving.
Players would make progress, leave for five or ten minutes, and when they came back, their profile had rolled back to an earlier state.
Fixing the issue took a few layers:
- First, we raised ZenControl's maximum memory back to what it was before.
- Second, we changed the cache so it no longer evicts data if writing to the database fails. Even if the database goes offline or rejects writes for any reason, the data now remains safely in memory until it can be saved successfully.
- Finally, we made the periodic save thread restart itself if it crashes.
But these changes still aren't enough. In the event of a ZenControl crash, there could still be a few minutes of rollback on SkyBlock profiles. That brings us to...
Why ZenControl is on its way out
If you're a software engineer, you're probably thinking our infrastructure isn't great, and I completely agree. The truth is we've been slowly migrating off this centralized ZenControl system for a while now. You can see this in our newer systems: Bazaar, Contests, Leaderboards, and others all run as their own services rather than relying on ZenControl, and that's intentional.We recognized that ZenControl was a problem long ago, and ever since we removed the original Auctions implementation (which lived inside ZenControl), we've been planning a path toward a more stable and modular architecture.
But this isn't an easy task, and the biggest part has to happen in a single pass: it requires rewriting every endpoint that currently modifies ZenControl's in-memory cached objects so they make direct database requests instead. Once ZenControl is a fully stateless service, we'll be able to make changes to it without taking the whole server down for maintenance.
Regional servers
Since we're already on the topic of server infrastructure, let's talk about a highly requested feature: regional servers.As mentioned above, some time ago we fully migrated our servers from a hybrid stack of multiple providers to a complete stack on OVHcloud with a heavy investment. This fixed the network issues that prevented servers from communicating with each other efficiently.
That said, for players far away from the server, like those in Asia, high ping is unavoidable because it ultimately comes down to the physical distance between you and the server. Internet infrastructure has improved a lot over the years, and current latencies are already close to the physical limit of what's achievable.
The real solution to this is regional servers, and yes, we want them too. But running multiple regions is not as simple as just "buying servers". Every region would require its own complete infrastructure: separate databases, backend services, game server instances, proxies, and everything else that makes the network run.
Building and maintaining all of that would require a massive investment, one that's simply beyond what we can support right now, especially with server costs having risen significantly in recent months due to the AI-driven demand for compute resources. The recent price increases are already putting additional pressure on our infrastructure costs, making an expansion of this scale unrealistic for the time being.
And that's before we even consider the code! CraftersMC SkyBlock, in its current state, would require a tremendous amount of work to properly support multiple regions. Being a single developer, this would mean putting updates on hold and working on this for potentially months. So as much as we want it, we can safely say it won't be happening anytime soon.You might be wondering "Most minigame networks have regional servers. What makes CraftersMC SkyBlock different?"
A typical minigame server is largely self-contained. Aside from things like friend join notifications, very little data needs to move in or out while a match is in progress. Most player data is simply submitted once the game ends. Because of that, even a 300 ms latency between the game server and its backend services usually isn't a major issue.
But for CraftersMC SkyBlock, it is a completely different story.
- Collected a stone? That goes to a service.
- Gained skill experience? That goes to a service.
- Want to visit a player's island? That goes to a service.
- Creating an auction? That goes to a service.
- In the middle of a Farming Contest, every single crop you break goes to a service.
Now imagine those services are on another continent. Instead of taking around 1 ms, every request suddenly takes roughly 150 ms. One request might not sound like much, but when thousands of these requests are happening every second, it's like a highway suddenly going from free-flowing traffic to a massive traffic jam. More requests arrive before the previous ones can finish, everything starts backing up, queues grow larger and larger, and what started as a small delay creates complete chaos.
That's why SkyBlock needs its own backend services and databases in every region. A minigame network can often get away with centralized services because each game is mostly self-contained. SkyBlock is different. Nearly everything you do depends on constant communication with backend services, so keeping those services physically close to the game servers is absolutely critical.
This is something the Hypixel team also discussed in their recent SkyBlock Q&A. Here's what they had to say about regional servers:
"We've had a lot of questions about Regional Servers and I have to sit down and give an answer, even if its outside of the scope of the Q&A.
The quick answer is no, this is not something that will happen. While it was something we considered many many many years ago, we've adapted to not having them and when it finally became a possibility, we were no longer in a position where it was worth the time and financial investment to make it happen. There are quite a lot to consider in making this possible but here's the few main blockers:
SkyBLock would need a lot of work in itself, which would require a portion of the team to work on for at least a year, causing a massive delay in content delivery. We are already dealing with a need for content to come out, especially on the combat front, and with us reaching Latest Minecraft content, suddenly there is already a slow down in content creation due to how much bigger updates now have to be.
Server costs are a lot higher than they used to be due to the volume we have today, when we originally had all those lofty dreams and ideals. The financial cost and burden of operating in a different data center would not be offset at all by the gains. This could push the company into making a really bad financial decision, causing us to have to shrink the team to account for it. It's just not worth it.
On top of that, adding a new region also means extra operating complexity, more DDoS mitigation, the need to explore, research and setup vendor relationships, more failure points, etc.
We understand latency is important for players, and we are constantly doing our best to optimize content as best as we can, but there's a limit to all of it. We are trying our best to push the current network to it's limit, but sadly, I have to reiterate - regional servers are not something that is currently on the table for Hypixel."
"We've had a lot of questions about Regional Servers and I have to sit down and give an answer, even if its outside of the scope of the Q&A.
The quick answer is no, this is not something that will happen. While it was something we considered many many many years ago, we've adapted to not having them and when it finally became a possibility, we were no longer in a position where it was worth the time and financial investment to make it happen. There are quite a lot to consider in making this possible but here's the few main blockers:
SkyBLock would need a lot of work in itself, which would require a portion of the team to work on for at least a year, causing a massive delay in content delivery. We are already dealing with a need for content to come out, especially on the combat front, and with us reaching Latest Minecraft content, suddenly there is already a slow down in content creation due to how much bigger updates now have to be.
Server costs are a lot higher than they used to be due to the volume we have today, when we originally had all those lofty dreams and ideals. The financial cost and burden of operating in a different data center would not be offset at all by the gains. This could push the company into making a really bad financial decision, causing us to have to shrink the team to account for it. It's just not worth it.
On top of that, adding a new region also means extra operating complexity, more DDoS mitigation, the need to explore, research and setup vendor relationships, more failure points, etc.
We understand latency is important for players, and we are constantly doing our best to optimize content as best as we can, but there's a limit to all of it. We are trying our best to push the current network to it's limit, but sadly, I have to reiterate - regional servers are not something that is currently on the table for Hypixel."
And that's all the nerd talk for today! I'm passing the mic to Phyruss for the less-nerdy stuff. Cya!
Hello! Phyruss here with a fancy new summer outfit.
Today I'll share some development footage from SkyBlock: End & Beyond, and give some sneak peeks for SkyBlock v0.5.1, which is just around the corner!
First, let's get behind the scenes of SkyBlock: End & Beyond!
Early building stages of The End
Setting up zones in The End using our chunk selection tool
First steps in zone.end
An alternate portal design to The End, scrapped later
Just like all the other pets, Ender Dragon pet also came to life as a Skeleton
The very first steps of equipment slots
While adding equipment slots to your precious inventory, we wanted to see how far we can go
Weird stuff kept happening as always
We lost track of a lot of dragons during our tests
Knockback causes unexpected Village visits
MMMMMMMMMMMMMMMM
Some Strong Dragon Fragments
Uhh, are you okay, Mr. Dragon?
Dragon refuses to die
"Talk to Softy" 
Ɇ₦ĐɆⱤ ₦ØĐɆ ⱧɄ₦₮ ₣₳łⱠɆĐ
A tiny amount of Ender Nodes
Scrapped Dragon Fragment textures
Scrapped Protector Dragon designs
Scrapped Superior Dragon designs
We even tried a chromatic design for it!
A scrapped dragon death animation where they crash into the ground
Drafts and Concept Arts for custom boss bars
Alternate Seer design
Alternate Ice Monarch Enderman design
SkyBlock: End & Beyond logo drafts
Prototype images of some of the new menus
A scrapped surreal keyart for the update
MS Paint drawing for the models of Dragon Fragments
Another MS Paint drawing, for the ranged damage indicator
Sus rune
Acacia Stop item texture 
Yeah, imagine
There were some balancing issues with dragons during testing...
The Teddy Bear sounds were actually recorded by us, from a cat toy
Making of the End & Beyond keyart
That's all for the stuff behind the scenes! Now let's talk about what's next.
SkyBlock v0.5.1
Unlike the previous updates, we've been releasing smaller changes as hotfixes ever since v0.5. That means the next update is SkyBlock v0.5.1, which is coming very soon!As the name says, while it's not a major update, it still packs a huge number of bug fixes, QoL improvements, and some new content you've all been waiting for!
It's finally happening!
As irrl mentioned, we've been adding most of the new blocks from recent updates for quite some time now. With SkyBlock v0.5.1, all of these blocks, along with a few custom ones, will be available in the Builder's Shop.New CraftersMC Originals textures
We have a new batch of textures getting added to CraftersMC Originals with SkyBlock v0.5.1. Here are the new looks of Blaze and Frozen Blaze Armors:One more thing...
I'd tell you what the biggest part of this update is... but I'd rather not awaken it just yet.Here's all you're getting for now:
And that's all for today! Thanks for taking the time to read. We have a lot more to share in the near future, so you won't have to wait long until you hear from us again. See you in the next one!
