Why No NetGame Object?

It’s hard to know what the best way to design some API is. I have no idea if the designs I picked for HappyFunTimes are good or bad or what. There are a few minor things I’d change if I started over, mostly cosmetic but otherwise I’m mostly happy with it.

But, a few days ago I decided to add a feature to allow multiple computers in a single game. Up until this point HappyFunTimes has been about 1 computer running the game and a bunch of phones as controllers but there’s a project I wanted to make that used multiple machines, in particular I wanted to make an extremely large platform game where you’d walk down a wall or hallway with say 10-15 meters of screens and take your character from one screen to the next. Each screen would be controlled by a separate computer running the game on that screen.

For that all I needed as the ability to move a player from computer to computer. I added that feature in a few minutes, tested it and it worked. Awesome!

I’d done some other projects across computers before and in every case I’d just have to have a special URL for each computer that specified which part of a larger virtual display that computer represented. For example the WebGL Aquarium does this Doing it that way is great if you already know what the layout and dimensions of your monitors will be but I’d always wanted to make a monitor layout editor similar to OSX and Windows. To do that I needed to enable games to talk to each other so that they could all inform at least one of them their dimensions and so that at least one of them could send commands to the others about what part of the large display they were supposed to represent.

And so I set out to write that.

My first thought was I’d have some kind of sendMsgToGame function that took the msg you want to send and an id for the game you wanted to send it to. It would be up to you either make up an id yourself, probably from a url, or the system could make one up for you.

But then I thought maybe, just like players have a NetPlayer object that is used to handle the communication between a player and the game I should have a NetGame object that does the same thing for games. This would mean, just like a player, there would be a ‘gameconnect’ message and you’d get passed a NetGame object. If you wanted to send a message to that game you could use someNetGame.sendCmd and anytime a game sent you a message it would arrive on that object.

someNetGame.addEventListener(
    'someMsgIExpectThisGameToSendMe', 
    handleThisTypeOfMsg);

I got that all working, wrote tests, made sure you could send and receive messages, that a disconnect message would get emitted as well.

But… Then the last thing I did was try to implement a broadcast function, broadcastToGames. I knew this was a useful function because I’d used it in the WebGL Aquarium demo I’d linked above. I’d put some camera settings sliders on one computer and broadcast them to all the computers. Broadcasting in this way meant you’d get the same message send back to yourself so you didn’t need custom code. All the machines received their settings from the network, even the one deciding the settings. This was a great way to propagate the settings to all the computers. But, when I tried to implement on top of NetGame it I ran into a problem.

To broadcast you’d call someGameServer.broadcastToGames but given the way NetGame worked all the other games would receive the message on the NetGame object that represented the computer that sent the message but, the computer that sent the message would get the message back on its GameServer object, in this example someGameServer.

That meant it wasn’t at all symmetrical. For the camera settings example in order to accept settings from any computer you’d have to write code like

// when a new game connects
someGameServer.addEventListener('gameconnect', function(netGame) {
  // setup a listener to handle setting the fov
  netGame.addEventListener('setCameraFov', setFov);
});

// Setup yet another listener for when we broadcast 
// a new fov setting because that will
// come back to us here
someGameServer.addEventListener('setCameraFov', setFov);

function setFOV(data) {
  camera.fov = data.fov;
}

That just smelled bad to me. Why should you have to set things up twice? I didn’t have to in previous games. Plus, you had to track all these NetGame objects.

I ended up getting rid of the NetGame objects. There is no gameconnect message. Instead, messages to a game from another game always arrive on the GameServer for that game. So previous example of setting the camera fov removes the second case

someGameServer.addEventListener('setCameraFov', function(data) {
  camera.fov = data.fov;
});

Much simpler.

If you want other games to know you connected broadcast a message to them

var server = new GameServer({id: "foo", allowMultipleGames: true});
server.broadcastToGames('imAlive');

The other games will receive the message along with your id.

// some other game
var server = new GameServer({id: "bar", allowMultipleGames: true});
...
server.addEventListener('imAlive', function(data, id) {
  console.log("some game just joined. Its id is " + id);
}

This will print

some game just joined. Its id is foo

Of course when you broadcast that message will come back to you so you might need to filter out your own id

var ourId = "foo";
var server = new GameServer({id: ourId, allowMultipleGames: true});
server.broadcastToGames('imAlive');
...
server.addEventListener('imAlive', function(data, id) {
  if (id != ourId) {
    console.log("some game just joined. Its id is " + id);
  }
}

Of course depending on your needs maybe you don’t want to filter. For example if you made a screen position editor like I referenced above ideally you’d just display the id of each machine with a rectangle to represent it. If you created those in your imAlive message then not filtering would automatically give you an entry for yourself.

If you don’t supply an id it will be provided on connect.

server.addEventListener('connect', function(data) {
  console.log("my id is: " + data.id);
});

That id is also available on the GameServer object but it is not available until it has connected.

Good:

var server = new GameServer();
server.addEventListener('connect', function(data) {
  console.log("my id is: " + server.id);
});

Bad:

var server = new GameServer();
console.log("my id is: " + server.id);  // ERROR! id not available yet as server has not connected.

Finally you need some way to know when a game disconnects so there’s a gamedisconnect message

server.addEventListener('gamedisconnect`, function(data, id) {
  console.log("Some other game " + id + " disconnected");
});

I don’t know if this post made any sense. It kind of ended up being like docs. Partly I just wanted to document the iterations. I spent a lot of time making the NetGame implementation and testing it only to throw it all way. Even after I threw it away I went through a few more smaller refactorings.

The first one was I made up a gameid command to pass back the gameid. I didn’t like that at all though. It took me a while to figure out how to pass it back on the connect message where it seemed like it belonged. Another thing that happened at the start was I had a 2 ideas per game. The generated id AND a user assigned id. It took a while to come back to just 1 id. Either the on you assign or the one you let the system assign for you.

I’m feeling relatively confident that what I ended up with is a good solution. I guess one more thing though is that I didn’t push out a version until I felt good about it. Sure there aren’t lots of devs yet but taking this stuff seriously will hopefully more confidence that I won’t have to make future breaking changes.

The Story of HappyFunTimes

I never intended to make HappyFunTimes what it is today. This is I guess an example of what happens when you just do something. Things happen.

It started off when I was working on Chrome. I learned about WebSockets. I might have seen some tutorial on how easy they are to use. I think I read about Socket.IO, a popular library that can use WebSockets and it just occured me, as it has several others, that it would be easy to use WebSockets to make certain kinds of games.

My first game was PowPow. You can see the commit history shows it was made around April 10, 2011. That’s 3 years before I started on HappyFunTimes. In fact the video linked to on that page is from April 8th. At the time I had people using their laptops as controllers. After that one day playing I tried making phone controllers. Being April 2011 it was still common for Android phones NOT to have multi-touch or to have broken multi-touch so if you go look at the code in that repo you’ll see the iPhone controls are Left/Right/Fire (3 fingers) but the Android controls are “point the direction you want to go, tap to fire” (1 finger)

At the time I thought it would be good to make a library but when you’re working full time, or when I am, it’s often hard to find motivation to do things outside of work. I suppose that’s kind of stupid since I made PowPow outside of work but it only took a few days. So, I guess I have no idea why I find motivation for some things and not others. If I had infinite time and infinite motivation there are sooo many things I’d work on.

As a side note, in 2012 I used the same tech for our Molyjam game, “Octopi Everything” which I plan to port to HappyFunTimes (if I can motivate myself :P)

Then last year in June 2013 I quit my job at Google. I traveled and goofed off and didn’t really know what I wanted to do. In March I found myself in Kyoto for Bitsummit. There I met Eddie Lee and Kalin of Funktronic Labs along with Sagar Patel, Andrew Palmer, and Edwon. Also in Tokyo I had previously met Alvin Phu and they were all so inspring. Everyone was working on something. Alvin in particular had a fulltime job with crazy Japanese overtime and still was working his ass off on his own game.

So, I started feeling like I should get off my ass and start working on something. I still had no idea what but I should at least spend time making something, anything. I thought about it and though, “ok, fine. I guess I’ll turn that code from powpow and octopi into a library finally and then maybe make a mobile game after”. I thought it would take at most a couple of weeks. I didn’t intend to make any money. I just thought it would be fun to put out the library and maybe have a game jam (which I still need to do!)

And, it did only take a couple of weeks at most. A few days to pull out the library. A few hours to make the first sample. But, if you’ve been to a game jam in the last few years or paying attention to indie dev everyone and their sister is using Unity3D. So, I felt like if I wanted to get any usage I needed a Unity3D version so you could make the games in Unity. Since the controllers have to be written in JavaScript I needed JSON support for Unity because JSON is how JavaScript communicates. There’s something called JSON.Net but it didn’t fit my use case. It’s a fine library I guess but in order to support the features I needed it required .NET 4.0 and Unity only supports .NET 2.0. So, I had to write my own. That was probably several days of work. As another aside I find it super frustrating to think “I’m going to do X and I’ll be done in a couple of hours” only to find that I first have to do “Y” which itself is going to take hours to days. I did get to learn some deep dark secrets of .NET though and with that working I reproduced the simple example in Unity

Next up I thought I should try dual stick controls. Partly because I wanted to make a unity example that shows walking a character around. So I hacked together ShootShoot.

With that working I got a unity character example working which didn’t take too long but took longer than expected dealing with UnityScript vs C# issues.

After that I wanted to see how a platformer would play so I threw together JumpJump and shortly after that I decided I wanted to make something showing off round based play among other things and wrote BoomBoom.

I ran into another issue which was I could not get consistent performance out of Chrome using the Canvas 2D api so I ended up writing JumpJump and BoomBoom in WebGL. That ended up making some useful libraries like a single quad single draw call tile map renderer as well as a runtime color adjustable sprite system.

In writing these games though things kept popping up. For example, originally you’d need to go to long and complex Urls. To play jumpjump for example you might open a browser and go to http://localhost:8080/examples/jumpjump/gameview.html then you’d get out your phone and go to http://192.168.2.9:8080/examples/jumpjump/index.html. That’s a lot to type, especially on a phone. Of course I used bookmarks and other things to make it go faster but eventually I got tired of typing so I thought, “hmm, I’ll write a script that generates an index.html that lists all the controllers”. I wrote that. I also wrote ones to list the games. Now I could go to http://localhost:8080/games.html to pick a game and controllers could go to http://192.168.2.9:8080/ and pick the corresponding game. That was much better. So much less typing.

But then, I realized if there’s only one game running why not just send the controller there automatically. So added a script to the page contollers go to that lists the games to check which games are running. If there is only one just go to it. This was awesome. Now I’d start a game, then on the phone I’d just go to http://192.168.2.9:8080 and it would automatically join the game! It seemed so awesome!

That led to the next step. Using the same code, when game disconnects the phones get notified so I used the same code that checks if any other games are running. If there is only 1 game running I jump to that game. The effect was that the system felt like a virtual console. I’d start it up, phones would connect, they’d jump into the game, and if I switched games they’d all jump to the new game. This was feeling really really great. There was even the added and unintended bonus that I reloaded the same game the controllers would see that as the one game running and reconnect. That was great for development. I’d tweak the game and click “reload” in the browser. That would have the effect of having all the controllers auto-reload. Faster iteration times FTW!

It still sucked though to ask people to go to their phone and type in something like http://192.168.2.9:8080. I knew about Captive Portals. They are those things you see when you use hotel WiFi or coffee shop WiFi and it asks you to log in inside the browser. If I could get HappyFunTimes to support something like that I could make it so when players connect to the WiFi they automatically connect to the game, no typing anything at all. It took a few days but I got it working. Note that feature only works on iOS and it only works if you configure your router a certain way to let HappyFunTimes respond to certain requires but the effect is pretty amazing. On Android some of the same code means if HappyFunTimes is setup this way users can just got to any Url to connect so I tell them go to to h.com or something like that and they connect.

These things are simple to explain and maybe it seems obvious but my impression is most people don’t really get it until they see it in action. I think even tech heads think “Yea, type some long address and you’re in” but when I actually demo HappyFunTimes in a room full of people and they see everyone effortlessly connect and play some games and they see me switch everyone to new games and go through all the demos it’s pretty impressive.

Announcing HappyFunTimes got a few people using it and I quickly realized I had some problems. All the games had to be checked into the same repo, or at least there was no easy way to separate individual games from the system. On top of that I had tried to show it at a certain event run by industry people. I sent them a link thinking “these are game developers, they won’t have a problem following the 5 steps to install to check it out”. Instead I got back a message something like “Wow, that looks hard”. Of course at the time I didn’t intend for end users or regular gamers to install it but gees!!, if an actual developer was bitching about installing node.js and cloning a git repo then there’s no way anyone else is ever going to get anywhere except for the most dedicated.

So, those two issues led to do things.

  1. I absolutely had to make an installer for at least Windows and OSX

    It had just install and work, no setup.

  2. I had to find a way to separate the games and make it easy for others to add their own games to the system.

Those two things led to several features. For the installer it wasn’t enough just to install. I also had to come up with a way to easily connect phones to the local game without having to configure your router and without having to type long cryptic Urls.

My first solution was to make happyfuntimes.net and write a script that scanned for the local happyfuntimes. Basically if you’re running Chrome on Android it could figure out your local ip address. Say 192.168.2.37. It would then ping every address on that network and see if it got the correct response. It worked! But, iOS Safari doesn’t have the feature to look up the local ip address. My next solution was if I can’t find out the local ip address then I’ll just guess. Most routers use a common set of addresses. I’ll just try them all. Unfortunately trying them all took > 5 minutes. I added some heuristics to guess better and therefore faster but it mostly didn’t work well.

Then the more obvious ideas was I should change happyfuntimes.net so when happyfuntimes starts it tells happyfuntimes.net its ip address. When a phone goes to happyfuntimes.net, happyfuntimes.net can see the matching ip address and tell the phone where to go. Problem solved!! Telling users to go to happyfuntimes.net is not as nice as auto-connecting but it’s not a cryptic Url so it’s easy to explain to users. Of course I wish is was just hft.com or something simularly short but short URLs are expensive 🙁

Next up was separating the games. npm, the node package manager, suggested a way to do it. It took a lot of trial and error though to finally get it working but the result is http://superhappyfuntimes.net and the related system.

The funny thing is though I never intended to take it this far. Each piece is just the next logical step as I built things up. I don’t know if there is anything to take away from that except that just doing stuff for the hell of it often leads places. Make some simple game or demo or app and you’ll likely end up with pieces of code or tech that you can use on the next thing or you’ll see how to tweak what you’re making into the something else and after a while you’ve made way more than you ever expected. I guess like they say, “Just do it!”

Why JavaScript

There are several reasons.

  1. It means no install required. Users can show up at some event and participate immediately.

    Potentially even other phones like Firefox OS phones or Windows phones or Xiaomi Phones should all work.

  2. It requires no approval from Apple.

    I’m happy that Apple seemingly does a good job of keeping malware off of iOS but I’m not so happy that they have to approve every app. Even if your app is not malware they might say no.

  3. It lets anyone make a custom controller.

    Theoretically you could use any language but Apple strikes again. Even if I decided to make an app Apple does not allow code to be downloaded into an iOS app except for JavaScript executed by Safari embedded into your app. So, even if I made an app, JavaScript is the only choice Apple allows. Of course you’re always free to transpile from some other language into JavaScript. With iOS8 now finally supporting WebGL that means in the near future the sky’s the limit on where to go with controllers in JavaScript! I’m looking forward to what people come up with.

I shipped!

superhappyfuntimes.net IS NOW LIVE!

Like I mentioned previously I’m pretty worried it can’t handle the traffic because it’s using meteor. I’d like to re-write it to just be as simple as possible but it’s time I don’t have at the moment so I’ll cross my fingers.

It’s version “super-alpha” which means I’m sure there’s 1000+ issues. You can seem some of them listed in the todo file in the repo. I’m sure I should be using issues or trello or something else to track todo’s but at the moment since it’s only me I find editing a text file a lot easier. If people start contributing I’ll have to move to something else.

But, that said, it does seem to work. The steps to run are probably less than obvious though. Listed below

  1. Download HFT and install it (see note below!)
  2. Run it
  3. Click the link for “superhappyfuntimes.net”
  4. Install a game
  5. Launch the game
  6. Make sure your phone and friends phones are on the SAME NETWORK as your computer.
  7. On the phone go to happyfuntimes.net

That’s it.

So many things to do and so many ways to make it smoother that I have to work on but it seems to work

Note: Please if you made a game migrate it to the new system so you can post it on the gallery.

NOTE: On Windows 8 (and 7?) Windows will block installation. Windows now requires your code to be signed. Signing costs $200-$500 or if you’re lucky you can get a free signature for open source project. I’ve applied but I haven’t heard back yet. Hopefully I get that issue solved. For now, after you download the installer, you’ll have to go to your Downloads folder and double click it. You’ll probably get a message something like

Windows Protection

Click “More” and you’ll see this.

Windows Warning

Click “Run Anyway”. After that you’ll have to find HappyFunTimes. On the start screen there’s a tiny message at the bottom

Windows New Apps

After that you’ll find it under H

Windows HappyFunTimes

If I’m lucky adding the signature I mentioned above will remove some of these steps.

Taking things for granted

I’m trying my best to SHIP the minimum viable product version of HappyFunTimes but it seems like every day I finish one thing only to find 2 more.

One of the things I learned early on is that many Unity devs are fairly new to many parts of software development. I keep forgetting that. One simple example is a friend wanted to use HappyFunTimes. He’s not an experienced programmer but he can hack stuff together in Unity. When he finds out he also needs to program the controllers in JavaScript and HTML5 he’s done. That’s not something he knows how to do. Of course he could learn how but he’s probably got a limited amount of time to get something done and learning JavaScript and HTML5 is not going to fit in that amount of time.

I have several ideas about that which I can list in another post.

Today though I was working on automatic uploading new versions of HappyFunTimes. There’s a lot of steps and it’s best to automate them. Since I expect lots of bugs it’s best I automate that early to avoid the frustration of updating and hopefully so I don’t forget a step and mess up.

One of the issues is keeping everything in sync. To make a release I need to

  1. Create an installer for Window
  2. Create an installer for Mac
  3. Check what the newest version of HappyFunTimes is online
  4. Check what version I have locally and make sure it’s newer than the one online.
  5. Check the versions of the installers match the version I expect them to be.
  6. Create a github release
  7. Upload the installers

At least that’s what I thought the steps were. One issue though is that when I make a release on github it takes a snapshot of all the files in the repo. It would nice of those files matched what’s in the installers. Especially because if you want to install in Linux I don’t have an installer so it would be nice if you could get the correct version of the files from github. In order to do that I’d need to check

  1. That my local git repo is clean.

    In other words I need to prove there are no changes sitting around not committed.

  2. That the local git hash matches the one one github

    This proves the files on github match the local files

So I checked if there’s an easy way to do that. I think there is. But … then I realize I should probably do the same thing when you publish a game. I’ve already written all the code to automate publishing a game although it doesn’t have that feature.

But, then I realize my code for publishing a game is all written using node.js and requires using git and github. I suspect more often than not Unity developers don’t know github nor git and I suspect most of them don’t have the time to learn.

I’m not sure what to do there. I think the system is automated enough that if you didn’t know git and github and all you wanted to do is publish a game you could do that by signing up with github, then typing one command and entering your github username / password. That would upload your game. That’s not really appropriate for github though. Github is only free if your game is open source which means you need to check in all your code at least.

Hmmm. I’m going to have to think about this. Maybe I could check the code in for you if you want? Or maybe MVP just means you’ve got to learn github and git and I can try to get that more automated later.

I know the devs that know git and github generally think “yea, do that. It’s easy” but having worked with less experienced devs I know there’s some big and frustrating hurdles there.

Fixing the wrong bug.

This is hilarious. I had Player, the server side object that tracks the connection to a smartphone for a single player, have a heartbeat ping because often players would just stop playing on their phone by having their browser go into the background. In such a case they aren’t disconnected from the server. So there’s this idle player in the game waiting for a networking message. Maybe “waiting” is the wrong word, rather it’s as though the player is making no input.

I wanted to remove those players so I have a heart beat. If no input from the player comes in 5 seconds I ping the player’s smartphone. If no message comes back in 1 second I kill the player because his browser is likely no longer active.

So, I’m trying to ship. I test Unity. When the Unity game quits the players are not disconnected. My (bad) intuition says “hmm, the socket must not be getting disconnected by Unity. I don’t remember this being a problem but I’ve almost always tested Unity from the editor instead of exported apps. I’m testing the apps. I try disconnecting it manually using OnApplicationExit. No change. I figure given that maybe because the C# websocket stuff I’m using is multi-threaded it must be that the disconnect not actually getting executed before the app quits.

Fine, I’ll add a heartbeat to the Game as well as the Player. The current heartbeat code is embedded directy in the Player object. I look at the code and see the Player is not deleted when it’s changed into a Game. I try refactoring the code so the Player‘s heartbeat just keeps going but I run into issues. Revert all that and decide to implement differenty. I’ll make the heartbeat code a separate class and have both the Player code and the Game code use it separately. Run into issues again and revert all that.

I figure that my heartbeat should go at a lower level than it was. It shouldn’t be at the Game / Player level it should be at a layer between WebSockets and that. I implement that. Spend 60-90 minutes debugging. It finally works.

I go back to my Unity sample and test again. Controllers still don’t get disconnected when the game exists even though I know the ping is working.

I finally realise the issue has nothing to do with disconnecting. That was working all along. The issue is there’s a flag the game can pass disconnectPlayersIfGameDisconnects. In JavaScript it has 3 values. undefined (the default), true and false. Just a couple of days ago I added it to C#/Unity. It defaults to false in C#! DOH!!!! Changed it to default to true.

All that work adding a ping at a lower level had nothing to do with anything. Works. 6-8 hours mostly wasted. Well, let’s hope that’s better anyway 😛

Picking the wrong Framework?

I struggled with deciding how to implement superhappyfuntimes the website.

There’s a million options from a LAMP stack to python, node.js, ruby, go, and the zillions of frameworks written on top of them. I was mostly sold on using a node.js since I like the idea of one language on both the client and server.

In some ways though superhappyfuntimes is a really simple site. All it needs is a database of games. At the moment there’s no logging in, no user accounts, etc. That might change. Ideally when you register a game it should be registered to you so you and only you have permission to edit it. But, seeking my minimum viable product (mvp) that stuff can wait. Still, it seemed like a good idea to pick a framework that supports that stuff so I can easily add it later.

At the same time, sometimes I thought maybe I should do it with no framework. Just checking an big .json file with an array of info from all the games. There’s not likely to be more than 100 games for a while. I can hope for more but realistically it’s going to stay small. But still, I’d need a way to edit that file. I didn’t want to have to edit it by hand anytime someone wanted to add a game. I could write tools to edit it but that’s basically rewriting all the stuff I’d get from a framework.

Ultimately I picked Meteor. It was really easy to get started, it just worked, no set up, and they even had deployment somewhat solved. So, it was easy to get things basically working.

But, you knew there was a but…Meteor is heavy. Each user that connects gets a not-too-small JavaScript app downloaded to their browser. That app then contacts the server to get database info, by default using WebSockets which itself is heavy. That might be fine if you’re trying to write a gmail clone but it’s not so fine if you’re trying to keep expenses small. I’m running superhappyfuntimes out of my pocket. I’d prefer if I only needed one small server and not a whole server farm. The pages superhappyfuntimes serves are for the most part static. They only need to be updated when a new game is added or updated, otherwise they’re the same for every user. I’m running on a single 512meg digital ocean server. I suspect that’s not enough for more than a few users with meteor whereas if I had some kind of static page server it would probably be enough for thousands of users?

So now the question. Do I throw meteor under the bus and re-write? Is there someway to leverage what’s there? Apparently meteor’s current solution to static pages so to run a full browser on the server (phantomjs) and capture the pages that area created. I’m not even sure that would work. Browsers are huge memory hogs. I’d probably have to run it offline, capture the pages, then upload the results. That means I’d be back to manually running it and no realtime update unless I setup yet another server.

Meteor is pretty awesome. Although I don’t have too much experience with other frameworks some cool things about meteor are that it’s default development environment is 100% live. Edit any .js, .html, or .css file and the moment you save it it updates. The server restarts, your page reloads, it’s seriously awesome! It also has a relatively nice templating system.

Maybe I should switch to another framework? I have no idea which one to switch to. Or maybe I should just pull apart meteor’s pieces and pair it down to just what I need? Or maybe I should start over from scratch? The public interfaces that update the game database are already handled it might actually be relatively simple to switch it all to custom code. I’ve no idea at the moment.

Sigh. I want to be working on games, not on infrastructure. Oh well. Live and learn. 😛

more npm sadness

I feel bad these last few posts are rants but …

I needed to make http json requests. I have a snippet of code I’ve been using for the last 4 years. It’s about 90 lines line. It’s for client side browser JavaScript. I needed something similar for server side node.js code. At first I thought I’d write my own. Having just submitted a pull request for the github package I’m generally familar with how to make a request. It seems pretty straight forward.

But…, I decide, well, in the spirit of npm I should see if there’s already a package for this. I quick search and request-json comes up. The API looks simple. How bad can it be? I take a brief look at the code. It’s in coffeescript. That’s a little scary as now I need a coffeescript compiler but what the heck, let’s give it a try.

$ npm install request-json --save
npm http GET https://registry.npmjs.org/request-json
npm http 200 https://registry.npmjs.org/request-json
npm http GET https://registry.npmjs.org/request-json/-/request-json-0.4.10.tgz
npm http 200 https://registry.npmjs.org/request-json/-/request-json-0.4.10.tgz
npm http GET https://registry.npmjs.org/request/2.34.0
npm http 200 https://registry.npmjs.org/request/2.34.0
npm http GET https://registry.npmjs.org/request/-/request-2.34.0.tgz
npm http 200 https://registry.npmjs.org/request/-/request-2.34.0.tgz
npm http GET https://registry.npmjs.org/qs
npm http GET https://registry.npmjs.org/json-stringify-safe
npm http GET https://registry.npmjs.org/node-uuid
npm http GET https://registry.npmjs.org/forever-agent
npm http GET https://registry.npmjs.org/tough-cookie
npm http GET https://registry.npmjs.org/form-data
npm http GET https://registry.npmjs.org/tunnel-agent
npm http GET https://registry.npmjs.org/http-signature
npm http GET https://registry.npmjs.org/aws-sign2
npm http GET https://registry.npmjs.org/oauth-sign
npm http GET https://registry.npmjs.org/hawk
npm http 304 https://registry.npmjs.org/json-stringify-safe
npm http 200 https://registry.npmjs.org/node-uuid
npm http 304 https://registry.npmjs.org/forever-agent
npm http 304 https://registry.npmjs.org/form-data
npm http 200 https://registry.npmjs.org/qs
npm http 304 https://registry.npmjs.org/http-signature
npm http 304 https://registry.npmjs.org/tunnel-agent
npm http 304 https://registry.npmjs.org/hawk
npm http 304 https://registry.npmjs.org/aws-sign2
npm http 304 https://registry.npmjs.org/tough-cookie
npm http 304 https://registry.npmjs.org/oauth-sign
npm http GET https://registry.npmjs.org/combined-stream
npm http GET https://registry.npmjs.org/async
npm http GET https://registry.npmjs.org/assert-plus/0.1.2
npm http GET https://registry.npmjs.org/ctype/0.5.2
npm http GET https://registry.npmjs.org/asn1/0.1.11
npm http GET https://registry.npmjs.org/punycode
npm http GET https://registry.npmjs.org/sntp
npm http GET https://registry.npmjs.org/boom
npm http GET https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/cryptiles
npm http 304 https://registry.npmjs.org/assert-plus/0.1.2
npm http 304 https://registry.npmjs.org/ctype/0.5.2
npm http 304 https://registry.npmjs.org/combined-stream
npm http 200 https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/sntp
npm http 304 https://registry.npmjs.org/asn1/0.1.11
npm http 304 https://registry.npmjs.org/cryptiles
npm http GET https://registry.npmjs.org/delayed-stream/0.0.5
npm http 200 https://registry.npmjs.org/boom
npm http 200 https://registry.npmjs.org/punycode
npm http 200 https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/punycode/-/punycode-1.3.0.tgz
npm http 304 https://registry.npmjs.org/delayed-stream/0.0.5
npm http 200 https://registry.npmjs.org/punycode/-/punycode-1.3.0.tgz

WTF!!!! Seriously?

npm uninstall request-json --save

Here’s the code I wrote.

"use strict";

var url = require('url');

var sendJSON = function(apiurl, obj, options, callback) {
  var options = JSON.parse(JSON.stringify(options));
  var parsedUrl = url.parse(apiurl);

  options.hostname = parsedUrl.hostname;
  options.port = parsedUrl.port;
  options.method = options.method || 'POST';
  options.headers = options.headers || {};
  options.path = parsedUrl.pathname;

  var body = JSON.stringify(obj);
  var headers = options.headers;

  headers["content-length"] = Buffer.byteLength(body, "utf8");
  headers["content-type"] = "application/json; charset=utf-8";

  var callCallback = function(err, res) {
    if (callback) {
      var cb = callback;
      callback = undefined;
      cb(err, res);
    }
  };

  var protocol = parsedUrl.protocol.substring(0, parsedUrl.protocol.length - 1);
  var req = require(protocol).request(options, function(res) {
    res.setEncoding("utf8");
    var data = "";
    res.on("data", function(chunk) {
      data += chunk;
    });
    res.on("error", function(err) {
      callCallback(err);
    });
    res.on("end", function() {
      if (res.statusCode == 301 ||
          res.statusCode == 302 ||
          res.statusCode == 307 ||
          res.statusCode == 308) {
        sendJSON(res.headers["location"].toString(), obj, options, callCallback);
      } else if (res.statusCode >= 400 && res.statusCode < 600 || res.statusCode < 10) {
        callCallback(new Error("StatusCode: " + res.statusCode + "\n" + data));
      } else {
        try {
          callCallback(null, JSON.parse(data));
        } catch (e) {
          e.content = data;
          callCallback(e);
        }
      }
    });
  });

  if (options.timeout) {
    req.setTimeout(options.timeout);
  }

  req.on("error", function(e) {
    callCallback(e.message);
  });

  req.on("timeout", function() {
    callCallback(new error.GatewayTimeout());
  });

  req.write(body);
  req.end();
};

exports.sendJSON = sendJSON;

less than 100 lines vs 25k lines. Why do I need 25k lines? Maybe it supports some things my 100 lines doesn’t? Authentication can be handled by setting options.auth to the appropriate thing. Headers can be set in options.headers. What else is there? Yea I see it supports streaming the body. That’s literally < 10 lines of code to add. I see stuff about cookies but cookies are sent in the headers which means you can add cookie support by setting headers. Maybe it’s a structure thing? I’d prefer a few separate libraries I combine rather than a library where someone else has already combine them. At least for something as seemingly low level as “make a request for some json”.

Maybe I’m looking at it the wrong way but it sure seems like something is wrong when I need 25k lines to do something that seems like it should take only a few.

npm is awesome and sucks at the same time.

npm is a node package manager. It’s pretty cool. You make a folder and type “npm init”. You answer a few questions and it makes a “package.json” file recording those answers. From then on you can install one of the 81000+ “packages” by typing “npm install packagename –save”. The “–save” part make it update the package.json file to record that your project needs the package you just installed. That means if you give a copy of your project to someone else they only need the parts you actually wrote. The rest will be downloaded for them.

It also helps you update packages to current versions etc..

So that sounds great in theory. The problem is finding anything useful and working. 81000 packages in a lot. How do you find the good ones? I have no idea.

For example I needed a library to zip and unzip files. I search for zip on npmjs.org and see there’s a package, adm-zip, with 19000+ stars. Clearly it’s a popular package so I install it and start writing my code.

The first problem they can zip up folders but you can’t easily choose any kind of filter? Want to zip up a folder but skip the “.git” or “.svn” folders or skip files that end in “.pyc” or “.o” or “.bak”. Too bad for you, write your own from scratch.

So I submitted a patch. It’s been 3 weeks and I haven’t heard a peep out of the maintainers. In fact even with 19000+ stars there’s been no commits in 3 months.

So, having written the filter I write some code to zip up a folder.  The very first set of files I build a zip from don’t unzip with the standard “unzip” program built into OSX. Seriously!? WTFBBQ? 19000+ people and it makes bad files? Lucky for me it was the first test I made. Imagine if I had not found that bug for months.

Looking at the code it leaves a lot to be desired. I often write crap code to but a zip library is not that hard to write. There aren’t lots of design decisions to make. It shouldn’t be that hard.

I go looking for other libraries. Results are mixed. Most of the libraries require the entire contents of the zip to be in memory either compressed or uncompressed. That’s arguably unacceptable. You can have multi-gigabyte zip files. You need to be able to stream them through the library. I looked for something along those lines. Didn’t find anything. Needed to get shit done so I’m using JSZip and Moxie-Zip and have a TODO in my notes to replace them with something that steams.

On top of lots of non-working packages there are packages that seem like they should be small that have way too many dependencies. For example I needed a prompt library since because of node’s event driven model that’s harder than non event driven languages. Sure I could roll my own but who knows what dragons lurk there. Better to use something where the dragons have already been slain.

So I do a search and the first thing that comes up is the “prompt package” with 5000+ stars. I install it and see it install 20+  dependences. Seriously? I need 20+ dependencies. That’s 186000 lines of code just to present a prompt?

  • Then I read the docs and I see this.
    var schema = {
      properties: {
        name: {
          pattern: /^[a-zA-Z\s\-]+$/,
          message: 'Name must be only letters, spaces, or dashes',
          required: true
        },
        password: {
          hidden: true
        }
      }
    };
    
    //
    // Start the prompt
    //
    prompt.start();
    
    //
    // Get two properties from the user: email, password
    //
    prompt.get(schema, function (err, result) {
      //
      // Log the results.
      //
      console.log('Command-line input received:');
      console.log('  name: ' + result.name);
      console.log('  password: ' + result.password);
    });
    

    Pretty easy right? The output from the above script is:

    $ node examples/property-prompt.js
    prompt: name: nodejitsu000
    error:  Invalid input for name
    error:  Name must be only letters, spaces, or dashes
    prompt: name: Nodejitsu Inc
    prompt: password:
    Command-line input received:
      name: Nodejitsu Inc
      password: some-password
    

That makes no sense. Object properties in JavaScript do not have a guaranteed order so there’s no reason to believe the above example would ask for “name” before “password”.

I think “oh, maybe their sorting the properties alphabetically” so I check the code. Nope! So, no confidence in this package 🙁  I did at least file an issue to point out the problem. So far no response.

Switch to the asks package. Hey, only 30000+ lines of code just to prompt the user?!? 🙁

Now before you think I’m just complaining I do try to fix things when I can. I needed to interface with github. I thought maybe I could find a library. I did a search and found the github package. Yea! Unfortunately the only API I needed was the Releases API and the github package had no support. Since I didn’t find any other solutions I just added it myself and submitted a pull request AND I GOT A RESPONSE! Yea, a package that is still actively maintained! Still waiting for a response to the second pull request.

To be fair this has nothing to do with npm which is awesome. It has to do with various libraries which are not. This is not unique to JavaScript or node.js. I’ve found lots of poorly written and bloated libraries in every language I’ve used. I appreciate that people are making their code available and that it solves people’s needs. I just kind of wish their was some trustworth curation about which packages actually work and have a quality code base.

Frustrations of making a web site

Tech gets out of data. Most of the websites I’ve made use a LAMP stack. Most of them are just static pages with some JavaScript or they’re wordpress based like this one.

For SuperHappyFunTimes I felt like I wanted to use node.js. SuperHappyFunTimes itself is mostly written in node.js and it seems like for the last 2-3 years it’s one of the major topics so I thought I should probably go that way.

On top of that I’ve never written a website with users and the ability to log in etc and I didn’t want to write my own since it’s easy to make a ton of mistakes. It seemed best to use some framework that handles all of this.

The first problem though is that ISPs that support node.js are far more expensive than LAMP based ISPs with no limit on how much money you’ll be charged.

This is arguably one of the reasons why PHP and LAMP stacks are still hugely popular. PHP’s design lets multiple sites run on the same server. Every other server tech except CGI requires a server per website. Sure, you can use virtual servers but virtual servers are still way more resource intensive than a shared LAMP stack.

So, that’s a little scary. I have no current monetary plans for HappyFunTimes so I’d prefer not to blow wads of my own money on it. (although I guess if I value my time I have blown wads of money on it :P)

Next up is frameworks. My original idea was to leverage npm.  It’s open source, it has a website and repo of “packages”. Rename “packages” to “games” and reskin the website and that seemed like enough.

Unfortunately I tried cloning the npm-www repo and bringing up its vagrant box. It failed.  The server it tries to clone from is gone apparently.

Then I realised npm was probably not the solution I wanted. I found this out when I saw a friend try to use HappyFunTimes. He downloaded it from github, typed “npm install” to install the dependences and saw it fail because someone deleted a repo. Not an experience I wanted to put end users through.

npm uses CouchDB so I spent a few days trying about CouchDB, walking through the official book. The tech seemed pretty awesome but I ran into several errors in the book. The book is open source so I submitted a patch to the first error I found but when I ran into more and realized some of the issues HAD NOT BEEN FIXED IN OVER 3 YEARS I lost all confidence in CouchDB. I’m sure it’s great tech but if they can’t be bother to update the docs it’s hard to believe it’s a good way to go.

Next up I tried different frameworks. Nodejitsu has some stuff. Again the docs and stuff seemed massively out of date. For example they push flatiron but as far as I can tell it’s dead. How can I have any faith in them if their own docs point to basically dead projects? So I gave up on that.

I found Meteor which certainly seemed nice. But…, it uses websockets which as far I can tell are resource hogs. Specifically a normal website you connect, it gives you page, the server and your computer then disconnect. This lets the server serve other people. WebSockets through keeps the connection open. This has the advantage that the server can send updates and other messages to the page for real time updates but it has the disadvantage that keeping that connection open requires lots of memory and there’s a limit number of connections a server can handle. For my needs I don’t need a realtime connection. The data on SuperHappyFunTimes is not currently massively changing and doesn’t need to be real time.

Apparently you can turn that feature off. I haven’t tried it yet. But I suppose that’s a minor issue. The bigger issue is these frameworks are scary. There’s no guarantee the teams making them will be around. You just have to pick one and hope for the best. I look for solutions, I see “todos” on their website that have been there for months or years, I see little activity on their repo. Are they still working on it?