Sneakerbots, Chrome Extensions, and the Evented Model
I like Air Jordan 1s. A lot. It's a new thing, especially since I hadn't owned a pair of shoes that wasn't all black since 1998, probably, but I'm enjoying this opportunity to step outside of my comfort zone and try something different.
Did you know that "sneakerbots" are a thing? Cause they are a thing. Limited edition sneakers are announced ahead of time and like any limited, premium item, there is a market around their buy and sell, so, naturally, a market has emerged around the very means to buy them in the first place. I watched a few YouTube videos and from what I can tell, "sneakerbots" look like applications that watch the clock and then zoom through the checkout process as soon as the shoes go on sale. The kicker is that few of them are free. Buyers often pay $25-$75 for a license that claims to work with all of the big retailers' websites.
I've been reading /r/sneakers a lot lately and there seems to be some misunderstanding about how they work. In particular, it seems like a lot of people have this idea that sneakerbots are some magic tools that guarantee a successful purchase, but the truth is that they are limited by the web sites' servers' abilities to process requests. This, ultimately, is the greatest weakness of these things because it means that no matter how good your script is, you're always at the mercy of the server's ability to handle traffic. At the same time, it stands to reason that traffic will ramp up in the milliseconds after a pair goes on sale, so the faster you can hit submit, the better your chance of completing a purchase.
Regardless of whether paid sneakerbots are worth it, the whole thing bugs me. It's fucked up that most new releases are only accessible through eBay at 3x or more their original price. It's fucked up that I can buy new releases from literally hundreds of different auctions for $500+ but I've not once seen a pair on someone's feet in Manhattan or Brooklyn. These new releases are going straight from the shelves to eBay and I'm guessing that most will never see concrete. The paid sneakerbot market adds insult to injury, exploiting the desire to catch a drop without being able to guarantee a sale. The videos I saw looked shady as shit: you're installing some closed source software built anonymously and designed to exploit shopping carts. It has access to your browser, your account, probably your credit card info. Fuck everything about that.
Enter FSB, my open-source sneaker buyer in the form of a Chrome extension. I had been looking for an excuse to learn a bit more about extensions anyway and if I can give an alternative to paid shady software, that's cool. (I guess it'd be better if it was a bit easier to use, but whatever.) My current version is shitty: it targets Footlocker's mobile site, it only works with shoes, and there might be a bug that's keeping the checkout button from being clicked. (In fairness, it's a bug caused by Footlocker's mobile site, a mismatched cert name that stops JS execution.) I threw it together in a bit more than a day and when I tried to use it the morning of the Shattered Backboards release, Footlocker's shitty servers were a show-stopper. At the end of this post, I'll talk about how it could be improved and what it would take to build a better sneakerbot. You can find the Github repo at https://github.com/subvertallchris/fsb.
Failure aside, the process of building a simple Chrome extension was fun and more valuable than a pair of sneakers would have been. The most interesting part of the process was figuring out how to create a concept of shared state using the extension's background page. I ended up with a pattern reminiscent of an evented model, something we've been using quite a bit at work with Ruby and RabbitMQ, but entirely in Javascript with the some help from the Chrome extensions API.
I'll walk through some of the things that took time to figure out.
Technical Overview
Start by reviewing the docs on extension setup, I won't cover that since there are many good tutorials out there for that.
My manifest.json likely demands more permissions than it. It uses a non-persistent background page to maintain state, listen for events, and fire off new pieces of the process as events occur. cartwatcher.js and cartcheckout.js both act as registration scripts. They ping the background page so it can record their tab id and, if needed, start a process when it discovers them.
Chrome's extensions API makes it very easy to manipulate CSS and start scripts, but if you want to, say, get the value of a field from a form and use it somewhere else, or make the clicking of a button on tab 1 cause a button on tab 2 to be clicked, it can be tricky. background.js contains the bulk of the code that solved this problem and, mentioned above, feels a lot like how we use RabbitMQ to kick off actions on servers at work.
Evented Programming for Dummies
Most programmers, myself included, typically approach tasks from a procedural perspective. We think in terms of sequence: first you do A, then B, then C, and so on. In the Ruby language, we have something like this:
def a_method do_this then_this finally_this end def do_this # some things happen end def then_this # some more things happen end def finally_this # final things happen end
An evented model decouples one action from another. Instead of defining the sequence, you create independent actions that announce their behavior through messages, or events. Each action can listen for and respond to others. Think of it as, "When Section A completes, it announces, 'SECTION_A_DONE'." Anything that is listening for "SECTION_A_DONE" will start working on its own. Section A neither knows nor cares what happens after it and those actions that start don't care what fired the message, they just know that it's their turn to do something. We also do away with the idea of a central function that ensures that Section A flows into Section B. At my day job, we use a server called RabbitMQ to manage this process and some programming languages, like Go, have messaging baked in as core concepts.
With FSB, I had a few different tasks happening independently of one another: I needed a cart open, I needed something else watching a timer and prepared to POST a form, I needed to listen for a checkout and complete the purchase when the cart was populated. There are some procedural elements to it but approaching it from an evented perspective let me think in terms of state, not process. This worked well with Chrome's Extensions API. Tabs are able to send messages to the persistent background page, and the background page can send messages back to tabs, but they can't communicate directly, so purely procedural programming wasn't possible. Messages were at the heart of the most complicated parts of the sneaker-buying event, but they also let me compartmentalize the process in a way that was easier to manage than it would have been otherwise, so building entirely around messages felt right.
Another nice feature of the background page: it can also inject arbitrary scripts into a page. These scripts execute in the context of the tab but with an important limitation: depending on how the site's scripts are defined, certain functions may not be visible. For instance, a page might have jQuery loaded, but my injected scripts might not be able to see $. In the case of FSB, this was a bit of a problem — I wasn't able to call their function to move onto the cart, so I had to use JavaScript to click the button that kicked off that process. At one point, I inject jQuery along with my script to perform an AJAX POST.
On FSB's Operation...
FSB targets footlocker.com's mobile site. Mobile sites tend to load faster than their desktop equivalents, so this seemed like the thing to pursue. I also noticed there's no CSRF token required, so you can add an item to your mobile cart from anywhere on the site without having to load the product page first! Dumb. Sadly, the mobile and desktop sites share a bit of code that causes some JS errors that screwed me up later on, but whatever...
So, with all that in mind, take a look at the code. It's very straightforward. I suggest installing the Chrome apps and extensions developer tool so you can monitor the background page. Examine each message received and sent through background.js and you'll see how the process works. Each message injects a script, the script performs a task and sends a message back to the background page, which is received and starts another event. Messages can contain data and message receipt can return a callback to the sender. This let me capture data from an active tab, send it to the background page for collection, and/or request information from elsewhere. In all cases, the background page listens and responds to requests.
This really is the pattern repeated throughout the whole process: script is executed, tab sends message that script has started, background page returns data or instructions based on the message sent from the tab, tab receives message and processes the script's body, tab sends message when complete, the process completes. Call and response.
The readme at https://github.com/subvertallchris/fsb walks through operation, but I suggest you monitor each page through Chrome's console throughout the process. Background page, too. Each step announces its status. It WILL try to complete the purchase right now so if you don't want that, comment out the click here. On the final page, the inspector was revealing some SSL problems — it looked like a requested resource didn't match its cert, so scripts would sometimes stop running. I'm sure there's a workaround but since my focus was on learning the extension, not being guaranteed to get a pair of sneakers, I didn't obsess over this detail.
Improving FSB
There are two key areas that need work.
First, the UI of the page tab sucks. If I was trying to release this, I'd want something a bit friendlier, with a date/time picker and better size selection. I'd also want a more robust management console so you could see what was scheduled and what was active.
Second, it needs to make multiple simultaneous attempts to work around the weakness of Footlocker's servers. You'd need to manage the global state as well as the state of each tab.
I don't plan on going any further with this unless someone wants to help out. It was a fun project and I'm happy for the experience but I'm not looking to get any deeper into this nonsense. The money I've been spending on shoes is bad enough but I value my time even more and there are enough demands on that already!
Model Caching in Rails, or when a Student is not a Student
For a few months, we’ve had a few reports by users from Neo4j.rb users of an odd bug. The story goes, “I try to create a relationship between two nodes but the type checking tells me that one of the nodes is not of the appropriate type but I know that it is.” In code, it could look like this:
student = Student.first
lesson = Lesson.first
# Creates a relationship between the lesson and student
lesson.students && student
More specifically, it was always reported by Rails users dealing with create actions. They’d be loading a node, creating a new node of another class, and associating the new node with the old.
student = Student.create(student_params)
lesson = Lesson.find(lesson_id)
lesson.students && student
An error would be raised saying words to the effect of, “Node type invalid. Expected
In my arrogance, I assumed that since the errors were not reported that often and I had never personally seen it, these guys must have had some conflict with another gem or messed up code somewhere. After all, if it was a real bug, we’d have seen it ourselves or our tests would have caught it, right? No, wrong — completely wrong. Finally, someone copied/pasted their code to me and I could see everything was done correctly. They also noticed that it only happened after updating a file and restarting their server would fix it. It began to sound like an issue with the Rails automatic reloader, so I set out to track it down.
Classes are Still Instances
The root, or maybe roots, of the problem turned out to be variables and constants hanging onto copies of classes, persisting across Rails reload!. There were two spots where this was happening, but to understand why it’s a problem, we need to talk about what a class is.
In Ruby, a class is an instance of Class. Because of that, it is possible that active Ruby process can have two independent instances of the same class, one a different version of the other. It’s something like this:
container = Container.new
container.enclosed_model = Student
# the Student model is changed, Rails `reload!` is called
# From this point out, our `Student` model can be thought of as Student v2.
# This is an instance of Student v2
student = Student.new
# This is false because our container's `enclosed_model` is Student v1
student.is_a?(container.enclosed_model)
# =>false
# And this hints at why
Student.object_id == container.enclosed_model.object_id
Comparing two objects’ object_id results is always the final answer on whether they are the same. By spitting out the object ids before the error was raised, I saw that, yes, Rails was loading a new instance of the class but somewhere in the bowels of Neo4j.rb was a reference to the old version.
After that, there were two questions remaining: where were references to old models hanging around and how could I keep things current?
The Culprits
The answer to the first question was pretty easy to track down but might be a bit too specific to Neo4j.rb to warrant inclusion in this blog post so I’ll just go over it briefly.
Exhibit A: Node Wrapping
When nodes are returned from the database, we have to figure out which model is responsible for it, and doing that requires us to find the model that has the same combination of labels as the node. It’s not a terribly expensive process but it can be wasteful since you only really need to perform that process once, when a combination of labels is first encountered, and should be able to reuse the results every time after. The results are stored in a hash, MODELS_FOR_LABELS_CACHE, which maps labels => model_constant. In a normal Ruby app, it’s not a big deal that this never gets GCed since you will only have so many possible entries; in Rails development mode, with models constantly being reloaded, that’s not always true, so it was possible for node’s to be instantiated using the wrong versions of models!
Exhibit B: Association models
The associations system and QueryProxy class are arguably the coolest features of Neo4j.rb. You’re able to define associations in models on the fly. Once defined, you can easily create and traverse relationships. It’s possible to build relatively complex queries that jump between models, executed lazily, as you do in ActiveRecord. What makes Neo4j.rb’s association chaining cool is that it will only perform one query, no matter how many models you jump across. So this:
student.lessons.teachers.children.lessons
…will find the lessons of children whose parents are teachers who teach classes taken by one student. In Cypher, it could look something like:
MATCH (s:Student)-[:ENROLLED_IN]->lesson)-[:TAUGHT_BY]->teacher)-[:PARENT_OF]->child)-[:ENROLLED_IN]->result) WHERE ID(s) = {student_id} RETURN result
In a model, the association definitions look like this:
has_many :out, :lessons, type: 'ENROLLED_IN'
That method returns an instance of Neo4j::ActiveNode::HasN::Association, which becomes bound to the model in its @associations hash. This association instance needs to know the model on the other side, so whether you let it infer that from the association name or use model_class to set it, it ends up with a @model_class instance variable that stores — you guessed it — the constant of the other class. When you try to create a relationship, it does a type check in Ruby. If the node given is not an instance of the model saved in @model_class, it raises an error. And there we have it: lesson.students && student will raise an error if student was not borne of the same version of Student held in its @model_class.
Whew! So… now what?
Learning to Let Go… of Models
The answer to the second question, how do we clean things up, was found just last night.
When I first diagnosed the problem, I was eager to get a workaround in so I patched the gem to not use is_a? to determine association compatibility, opting instead to use the node’s labels compared to the model’s labels. This solved the immediate problem but it wasn’t a real fix. Last night, someone commented to an issue on the devise-neo4j library that I had forgotten about, describing the same problem, and I realized that there’s a very good chance this caching was the root. He had done some research into Devise and posted a snippet that included a reference to ActiveSupport::Dependencies::ClassCache, so I looked that up and found a note about the before_remove_const method.
before_remove_const seems to be the solution. When implemented as a class method, it is called by Rails reloader at the start of a reload cycle. I was able to use it to wipe out the constants that were hanging onto models and trigger a refresh of @model_class in each association. You can see the PR here. I say it “seems” to be the solution because I’m still waiting on confirmation of the devise-neo4j issue’s resolution, but I’m reasonably confident. Even if it doesn’t, I think we’ve confirmed that there’s an old reference to a model hanging out somewhere, so we just have to figure out what we missed and queue it for update later on.
So there you have it! An interesting bug squashed and in the process, we saw more proof of Ruby’s “everything-is-an-object” ethos. We learned a bit more about ActiveSupport, some best practices when caching class names, and a crucial reminder to take bug reports seriously, even if they seem impossible to you.
Sublime to RubyMine to RubyMine + Vim to...?
As glamorous as the life of a co-contributor to a sort-of-kind-of-well-respected open source library is, I think one of the best perks is being able to cash in on the free, open source license of RubyMine. It took a bit of coaxing, mostly in the form of a lot of positive feedback from /r/ruby, and I was hooked after a few weeks.
A few weeks ago, though, I became mildly envious of my friends' at work who flew around the screens with Vim. I love keyboard shortcuts, always have, and the potential to make the editing process significantly faster was really attractive. Still, I love RubyMine's features, so... what was I to do?
Enter IdeaVim, the Vim emulator for RubyMine. Hallelujah, I have been saved! Real fast, I'll run through what I like best about this union.
RubyMine features I can't live without
- CMD + Click on any method to go to its definition, even if it's in a dependency.
- Right-click and there's an option to find all usages of a method.
- Highlight and CMD + Option + M to extract code into a new method. It finds variables and makes them arguments. Fuuuuck! So cool.
- YARD integration. If you annotate your methods, it will use the info to improve its code completion.
- The best find/replace/rename options I've ever encountered.
- Method calls get pointed out if I send the wrong number of arguments.
I know that's not much — it's sure as hell nowhere close to taking advantage of everything RubyMine offers — but they're all huge, huge game-changers. The thought of not having the ability to locate method source is brutal, I think it's a crucial option for anyone working with large codebases.
There are a few other features that are cool but just not for me. I use the test support on occasion but tend to want to run the same one or two specs repeatedly, not an entire file, so it's usually overkill. As for git, I find the CLI more comfortable, though its integration does seem fantastic. Finally, the Rails features are extremely cool but I rarely work with it, so they're lost on me.
Vim makes things faster
A lot of Vim users go crazy with plugins. It seems only natural, since every itch seems to be scratched by something that's very focused and sounds very helpful, but all that power comes with a steep learning curve. I think I'm mitigating that by focusing on the basics: fast movement around the screen and word/line deletion/replacement. You don't realize how much time you spend moving around the screen with some combination of mouse + arrow keys + ctrl/home/end until you start using EasyMotion (emacsIDEAs) in my case and get a hang of basic editing.
My Vim list:
-
emacsIDEAs is insane. I have it mapped to (CMD + J)(CMD + F)(char I want to find). It highlights that character all over the screen, replaces it with a character (a thru z) and then I hit that character to jump there on the page. Stolen from Google:
It takes a little getting used to but it's very precise, very helpful.
- Everything here. Everything.
That's it so far. It might not seem revolutionary but it's one of those "you-have-to-see-it-to-understand" things.
To those getting started with Vim + RubyMine...
I had some problems adjusting and still find some things a little annoying. My quick bullet-points for those who are new:
- Remap the hotkey that switches from insert to normal mode. I went with (CMD + j)(CMD + j). For some reason, I always have to hit another key — any key — after switching modes for it to register the change. Having to reach up to escape constantly was a real buzzkill.
- Remap emacsIDEAs basic word search. I went with (CMD + j)(CMD + f) and remember it like "jump" + "find". It's helpful because those two characters are always accessible.
- Don't be afraid to turn off Vim mode if you're frustrated. The nice thing about the plugin is that you still have RubyMine backing you, so it's not like you're going back to Notepad in its absence.
In Conclusion
I imagine that if this post is read by anyone, it will be people considering the switch to either or both. If that describes you, my advice is to go for it but go slow, don't feel pressured to use every new feature at all once. While the Vim Master Race might push you to use ALL THE PLUGINS ALL THE TIME, I find that it's such a huge paradigm shift from a traditional text editor that a slow ramping up is helpful. Find the tools that are easiest to work into your workflow, gradually increase resistance. Right away, you'll find an improvement in your work, and in time, things will only get better!
For my next trick, I intend to get better with Vim away from RubyMine. I've been doing a bit with Rust lately and find the switch back to SublimeText like... barely a step up from Notepad, frankly, but I'm still struggling with folder navigation and window management in MacVim. It's a process, I'll get there.
Happy coding!
A Simple API Request/Response Decision Tree
I threw this together for work. I think it’s easy to get lost when designing APIs, web/software/whatever, but keeping this in mind can help you stay focused.
A Java Crash Course for Ruby Developers, Part 1
(Part 1 of a multi-part series. Introduction.)
Welcome! Let's get right to it. This is going to be largely unstructured and reflects the current state of my Java experience, which is very little. It will hopefully help other devs who are getting started. I expect that:
- You know some basics about Java
- You know Ruby basics
I'm not going to mention many things that should be obvious, like... lines are terminated with semi-colons, Java is not sensitive to whitespace, etc,...
Let's do this.
#### Use an IDE for Development.While you can use a text editor to write Java, every resource I encountered referenced some IDE or another. The two big ones seem to be Eclipse and NetBeans. I ended up with NetBeans, though I honestly can't remember why. Having just switched from SublimeText to RubyMine, this was a pretty easy transition for me, as NetBeans feels a hell of a lot like RM, but it might take some getting used to for you. (Incidentally, JetBrains are the developers of IntelliJ, a Java IDE that I imagine is pretty amazing. It's also not free, so...)
If this is your first time using an IDE, the key features that I can no longer live without are:
- Command + Click on a method to find its definition.
- Right-click a method and click "Find Usages" to find everywhere that it's used
- Highlight, right-click, "Refactor" menu to easily move pieces of code into methods, new classes, sub/parent classes, etc,...
- Built in test integration and compilation
- Easy dependency management
- Flagging unused variables and dependencies
- Hints RE: object types required during method calls or class initialization
Focus on that and you'll adjust in no time.
Don't start from nothing. Find an open source project to modify.
Starting a new language can be daunting, especially something like Java that introduces some concepts that are very different from Ruby. I set my sights on GraphAware's TimeTree plugin for Neo4j, found where the public API endpoints were exposed, copied the methods, and started modifying from there. Anyone who learned HTML in the days before CodeAcademy and sites like it will probably have done this at some point.
Plan on writing tests.
Because Java is compiled, it's very annoying to test your code by poking at it in your app. Write small methods, test carefully, and you'll save yourself a lot of time. There's JUnit, which uses methods and simple assert methods to test behavior (sound familiar?) and there's also Cucumber-JVM, if you're more comfortable there.
Class Definitions
One of the hardest parts of figuring out Java for me was just reading the damn code. You should already know that Java is strongly typed, but... final? Static? Void? WTF? This ends up being very easy.
Basic class Signature Examples
The typical Java class signature looks like this:
public class MyClass {}
That makes sense, right? public, protected, or private define visibility, just like Ruby. The rest is self-explanatory. Java has single inheritance and uses extends, so:
public class MyClass extends AnotherClass {}
Interfaces
Instead of modules, Java has interfaces. An interface lists methods that must be implemented in the class. It deals with what is there but not how they will behave. (NOTE: In Java 8, you can define default behavior for interface methods. This helps you out in the event you want to add a method to an interface that is widely implemented.) A class that includes an interface looks like this:
public class MyClass implements MyInterface {}
At that point, any methods defined in MyInterface are expected to also be defined in MyClass. If they aren't and default behavior isn't provided in the interface, your code will not compile.
Interfaces are extremely helpful because when it comes to method signatures, you can say that a method expects or returns a MyInterface instead of MyClass. It would be like doing this in Ruby:
module MyModule; end class MyClass extend MyModule end # Will return true MyClass.new.is_a?(MyModule)
Ruby doesn't work that way; instead, we'd use respond_to?, but Java's way gets us to the same place by a different route: we are checking that a given object has specific behavior implemented. We care less about the object and more about what it can or cannot do.
There's a lot more to interfaces than I'm going to get into now, but this is a good start.
There is no initialize method, but...
Instead of initialize, you define a method of the same name as the class and do not include a return value.
public class MyClass {
public AnotherClass myVar;
public MyClass(AnotherClass myVar) {
this.myVar = myVar;
}
}
That means that MyClass is initialized with one argument, an object of type AnotherClass, that will be referred to within its init method as myVar. Once instantiated, myObj.myVar will be set to this argument. For example:
# Define the variable MyClass myObj; # Instantiate the object myObj = new MyClass(myOtherObj); # call `myVar`, will return `myOtherObj` return myObj.myVar;
There can be multiple constructors for a class
Java makes it possible to instantiate a new object using many different arguments, just define multiple constructor methods.
public class MyClass {
public MyClass(AnotherClass myVar) {
# do something
}
public MyClass(DifferentClass myMar, AnotherClass myOtherVar) {
# do something
}
}
Of course, the constructor methods would do something with those vars. Point is, you have flexibility in how you instantiate objects. This same rule holds true for methods.
Method definitions
Basic Method Signature Examples
Method signatures can omit visibility and will default to public but the best practice seems to always include it. They look like this:
public ReturnType methodName(FirstArgClass firstVar, SecondArgClass secondVar) {
# Some code
return # something here
}
Note that every method identifies not only what types of objects it expects but what it returns. To define a method that does not return anything, use the void keyword.
public void ReturnType methodName() {
# do something but do not call `return`!
}
To define a class method, use the static keyword.
public static ReturnType methodName() { return null; }
Method arguments are typed
Method arguments follow the simple pattern of ObjectType variableName, separated by commas.
public String myMethod(String firstVar, int secondVar) {
return firstVar;
}
There's not much more to it than that.
Variables
Variables are declared with types
This shouldn't have to be said at this point but:
String myVar;
That would allocate memory for a new String with name myVar.
Declarations can be assigned immediately... but this does not seem to be best practice
You can do this:
# assume `thisOtherMethod` returns something of type MyObj... MyObj var1 = thisOtherMethod(anotherVar);
My IDE corrects me when I do this, though, so I get the sense that it's a best practice to declare first, then assign.
Instance variables
Define instance variables in the body of your class.
public class MyClass {
public String myString;
public List myList;
}
Those can now be assigned within methods. This is the equivalent of @var in Ruby.
Class variables
Define class variables using the static keyword in the body of the class.
public class MyClass {
public static String myString;
}
Constants
Define constants using both static and final.
public class MyClass {
private static final String myString = 'oh hai';
}
The final keyword indicates a value that cannot be changed.
That's it for now. Keep an eye out for part two. Hope this helps someone out.