Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

Welcome to our site

Take a moment to join our board

Sign in to follow this  
Spirited

Chimera: Conquer Online Private Server Project

Recommended Posts

Pretext

Hi everyone. I decided a few months back, after the chaos of moving jobs and finishing schooling, that I'd get back into my hobby projects. You've probably seen some of that on discord, like my TV case project (which I'll post here soon, too) or my Conquer Online portal editor in Go, but the biggest project I have is still largely up in the air. Chimera is my Conquer Online private server idea, and it's gone through a few iterations of designs. Before I ask for feedback, here's a quick introduction to the project (ripped from its readme file).

Introduction

Project Chimera is a four-role open distributed server architecture, named after the mythical four-headed beast. The game service is composed of an authentication server, gateway server, realm server, and fog of region servers. As players connect using the Conquer Online game client, their login will first be validated against the authentication server. Then, their client will be redirected to the world's gateway server, which decrypts and routes packets to and from the realm server and active region server. The region server services the player using a vertical slice of the game world, and the realm server instructs the region server on which slice it hosts.

The separation of roles into individual, independent servers allows Chimera to experiment with rolling upgrades and server maintenance without interrupting players. Its distributed architecture allows for the game service to accept a large amount of players concurrently, while also delivering feature rich maps and monster AI.

Diagram

Here's a quick diagram I came up with for my own use in Visio.

architecture.png

Discussion

So, I found out recently that the way I was doing error handling in Go was wrong. I adjusted that and returned back to my original source in Go. I laid out all the project structure, figured out how packages will relate to one another, the interfaces I'll use for mapping everything together across all projects, etc. I felt like Go was a good starting choice. It introduces a lot of risk, but this isn't exactly a business project... so I thought I'd give it a try. In my experience, Go has a much better time handling request volume than C# does (comparing the async-await model in C# to goroutines in Go). Go also has a lot of language support for security, including a really solid big number implementation, and great networking support. C# is my fall back, but I'm going to try with Go first.

My main concern is the architecture. I've been trying to poke holes in it, and so far felt pretty confident in it. I'm not taking the project too seriously, but I do plan on running these servers on Raspberry Pis and Pi Zeroes to better exacerbate performance issues during development (and just for fun to toy with different docker deployment models). But in general, what are people's thoughts here? At the end of the day, if something does go wrong, I can at least share that experience here - but I'd hope it'd go well. Thanks for reading.

Share this post


Link to post
Share on other sites

It looks like there is a large number of internal communication. Are you concerned with latency issues? Also, it seems that every module is dependent on other modules, how will you handle the reliability of modules?  I would imagine some clever heathchecks with k8n would be a good place to start. 

Share this post


Link to post
Share on other sites
9 hours ago, Smallxmac said:

It looks like there is a large number of internal communication. Are you concerned with latency issues? Also, it seems that every module is dependent on other modules, how will you handle the reliability of modules?  I would imagine some clever heathchecks with k8n would be a good place to start. 

I'm not concerned with latency at all; the difference on an internal network is insignificant. I suppose you can say that modules are "dependent" on other modules since they communicate with them, but I'm not sure what you mean by how I handle reliability. Having this sort of distributed system aims to increase reliability (no longer a single point of failure if the game server crashes). In general, most actions can be completed by the region server without the realm server. The region servers are design to run on their own and handle realm server restarts and faults using the gateway. If it's down for too long, then an automated server maintenance could kick in, I guess. RPC between servers is over TCP right now, so it's not like I really have to think about health checks. TCP includes heartbeats out-of-the-box. In the future, I could add alerts and fault tolerance, but it's not really my main concern right now. I think that answers everything you asked me? If not, feel free to follow up.

Share this post


Link to post
Share on other sites

How will you determine the scalability or what metrics would require a new realm node to spawn? 

Share this post


Link to post
Share on other sites
44 minutes ago, Smallxmac said:

How will you determine the scalability or what metrics would require a new realm node to spawn? 

This isn't exactly something I've completely flushed out yet, but it could be the resources in use such as CPU and RAM, could be the amount of players connected to a server. Whatever the bottleneck is and however that can be best anticipated.

Share this post


Link to post
Share on other sites

So, getting closer to figuring out all the details. I decided to look into Kubernetes for scaling and orchestration of services. In terms of language, I'm settling on Golang and I wrote a small article about why for those interested. Looking forward to having more to show, but this is it for now. Cheers.

Read more: https://spirited.io/2019/04/21/an-evaluation-of-golang-for-game-server-development/

Share this post


Link to post
Share on other sites

I like your approach. You follow some of the ideas that I derived when I was prototyping a scalable environment. One of the biggest concerns I had was the number of entities on a particular map. One thing I thought about was chunking regions and having nodes to handle that, but you would then just scale the issue lower. For example, the bottleneck would then be on the number of players in a particular chuck. You would also have to handle cross chunk communication and they are visible. Another idea I had was possibly scaling channels. For example, if too many people are in TC and it proves to be a bottleneck it might be an option to start a new TC map and move players to that instance. The main problem I found with that is separating the players, in some cases that might not be a possibility (GW). This is just all assuming there was a bottleneck to begin with, this is all just for thought really. What are your ideas on such an issue? If said issue is present, how would you overcome it?  

Share this post


Link to post
Share on other sites
17 hours ago, Smallxmac said:

I like your approach. You follow some of the ideas that I derived when I was prototyping a scalable environment. One of the biggest concerns I had was the number of entities on a particular map. One thing I thought about was chunking regions and having nodes to handle that, but you would then just scale the issue lower. For example, the bottleneck would then be on the number of players in a particular chuck. You would also have to handle cross chunk communication and they are visible. Another idea I had was possibly scaling channels. For example, if too many people are in TC and it proves to be a bottleneck it might be an option to start a new TC map and move players to that instance. The main problem I found with that is separating the players, in some cases that might not be a possibility (GW). This is just all assuming there was a bottleneck to begin with, this is all just for thought really. What are your ideas on such an issue? If said issue is present, how would you overcome it?  

Good questions - Honestly with a system like this, something is always going to be a bottleneck. For Chimera, the minimum I'll process on a single node is one full map. I have no intentions of chunking maps beyond that point. The design complexity, communication overhead, and additional latency wouldn't make chunking across sections of a single map worth it. The way I design my nodes is such that every region service is essentially it's own lightweight game server, and those services all run in memory without database access. The server would easily be able to handle lots of players on a single map; I'd probably hit a self-made player limit just to prevent client bricking before hitting a performance issue. In terms of world scale, if I need more nodes to process maps, then I can add more nodes or increase resources (whichever is more cost effective in that scenario). I can also automate scaling to optimize costs or distribute work on heavy load, but let's be real, this is Conquer Online. I don't think it'd ever go over the minimum. 

Share this post


Link to post
Share on other sites
On 4/26/2019 at 5:02 AM, Spirited said:

but let's be real, this is Conquer Online. I don't think it'd ever go over the minimum.

Ah yes exactly, I just thought it would be a fun thought exercise if anything. This entire project, in general, is just overkill, to be honest, but it is fun none the less. I completely agree that this is an awesome exercise in a hard distributed computing problem that should hit all the hard parts of communication, data syncing and scaling. 

Share this post


Link to post
Share on other sites
11 hours ago, Smallxmac said:

Ah yes exactly, I just thought it would be a fun thought exercise if anything. This entire project, in general, is just overkill, to be honest, but it is fun none the less. I completely agree that this is an awesome exercise in a hard distributed computing problem that should hit all the hard parts of communication, data syncing and scaling. 

Well, in the case of a real MMORPG struggling to keep up with growth, all current MMORPG servers can't scale a single map beyond a certain point. There's a technical and game quality reason to introducing server player caps. There are reasons why MMORPGs split their playerbase across realms or continents. At some point, you just can't keep accepting players, whether that's a server resource issue or client performance issue. If we want to talk experimental, you could make Kubernetes nodes completely stateless, and instead store state across all servers in Google Memorystore or Cloud SQL (which returns queries in microseconds to single digit milliseconds). If you need more nodes processing packets from players, you simply spin up more nodes. It would be expensive, but it would work... in theory. It just wouldn't scale well with smaller to medium/large loads. For most cases, the model I chose for Chimera would be a "best fit". I mean, all players running around on the same map at planet scale would freeze any computer on earth, anyways.

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

By using this site, you agree to our Terms of Use.