Our Blog.

Expert thoughts from our software developers and design leaders.

  • Larry Morales

    How Emergent Design Solved N+1 Problem In Buffalo Pop

    Disclaimer. Those who are not familiarized with N+1 problem, you can check “What is the N+1 selects problem in ORM(Object-Relational Mapping) with details explaining the concept.

    Since associations were introduced in Pop, the N+1 problem became part of it: many queries hit the database in order to load whole model’s associations. Good news is N+1 problem is not a disease without remedy. In this post I will give you some steps I took to solve it by incorporating emergent design.

    Step 1: Start Coding By Example

    Coding by example helps you to define what’s your input and what’s your output. Code By Example means you define a scenario where  your solution is tested and what is the expected result. The method I used to apply this technique was Test Driven Development (TDD).

    I started to code by example for the most simple scenario I could think of. To me, the most simple scenario was to solve N+1 problem for the has_many type association.

    =========== Scenario 1 =============================
    
     Given I have 3 Users with one Book each one related
     Then I would expect the query performed to be:
    
     select books.* from books where user_id in ($1, $2, $3)
    

    This, translated into code, would be:

      // pop_test.go
      type User struct {
         ...
         Books []Book `has_many: “books”` 
      }
    
      // pop_test.go
      type Book struct {
         ...
         UserID int `db:”user_id”` 
      }
    
    // preload_associations_test.go
    func TestHasManyNPlusOneSolution(t *testing.T) {
       ...
    
      for i:= 1; i <= 3; i++ {
          user := User{}
          tx.Create(&user)
          
          tx.Create(&Book{
              UserID: user.ID,
          })
       }
    
      Debug = true
    
      users := []User{}
      preload(tx, &users)
    
      a.Equal(3, len(users))
      a.Equal(1, len(users[0].Books)
      a.Equal(users[0].ID, users[0].Books[0].UserID)
    }
    
    // preload_association.go
    func preload(tx *Connection, model []User) error {
    	return nil
    }
    

    Step 2: Be Explicit With Your Input And Output.

    As soon as I wrote down the scenario, I noticed there are three things that I needed to do in order to produce the desire output:

    • First thing is to iterate over every user and store its ID value in a slice.

    • Second thing is that user_id column in the expected query result. I need to pull it out from db tag in Book model.

    • Third thing is to link all books with their respective users once the query is successful.

    Because I’m in a premature start point, and I don’t want to go into the code details about how to read that field tag, In order to make that test to pass as quickly as possible, I hard-coded the solution and be explicit to have that test with a green face.

    func preload(tx *Connection, model []User) error {
        // 1) fill in ids.
        ids := []interface{}
        for _, u := range model {
            ids = append(ids, u.ID)
        }
        
        // 2) load associations.
        books := []books{}
        tx.Where(“user_id in (?)”, ids).All(&books)
        
        // 3) fill in model with associations.
        for i := range model {
            for _, book := range books {
                if book.UserID == model[i].ID {
                    model[i].Books = append(model[i].Books, book)
                }
            }
        }
    }
    
    

    Step 3: Go From Specific To Generic.

    Once I have green light and test is passing, I noticed the solution was composed into a three steps process:

    1. fill in ids.
    2. load associations.
    3. fill in model with associations.

    At this stage I need to be less specialized and be more generic with the solution. The reason is this moment preload function only knows how to preload users and books, but models can be from different types.

    In order to achieve generalization, I take one step at the time and apply some refactorings techniques until I’ve satisfied the desired generic level.

    I started to refactor step one (fill in ids). In this step, the generic level I want to achieve is to get a list of all ids value from model, no matter what type that model is. This pushed me to incorporate some reflections code and go a little deep in details in order to get the expected result. For every tiny change I applied to my code, I ran my test making sure is still getting green.

    Once I got the level of abstraction I want for my step one, I move on to step two (load associations) and do same thing. I stablish first wich is the generic level I want to go, and then apply all code changes and refactorings necessary to achieve the correct result. Once I’m done with step two I then move on to step three, always trying to achieve generalization. Here is the result of applying this process to the has_many preload function.

    Step 4: Repeat, Repeat and Repeat.

    Once I feel I’m done with has_many type association, I start to cycle again following same process for has_one type association. I repeat same three steps: Code By Example, Be Explicit and Be Generic (from now on CBB) until I got all four type associations covered. https://github.com/gobuffalo/pop/blob/nplus1/preload_associations.go#L158

    Wrapping up

    As software developers we tend to find code solutions based on our experience in one project and try to reuse them in other projects where we think they’ll match up. Sometimes, there are cases where uncertainty is a variable, and applying CBB process helped me to overcome that. I’ve testified the good impact this process has. The way that solution design emerges, without a deliberated thoughtful specification in mind. It feels like an organic living growing from its seed. Give it a try and let me know how it works for you.

    Thank you for reading this post. I hope you have enjoyed how emergent design helped me  to find solutions to N+1 problem in Buffalo/Go, and hope you apply it into your projects.

    Continue Reading
  • Larry Morales

    Code review: Refactoring (part II)

    In part one I talked about what refactoring is and all stuff related to that. In this post I want to show you how refactorings are related to frameworks and give you a real example of their use.

    Frameworks were design to make your life easier in some way, and they should fit to solve your problems. Nowadays developers rarely build apps from scratch, we use frameworks according to our needs to speed up construction, testing and delivery. You realize that get used to a framework doesn’t mean you are except from live without code smells. You can actually apply refactoring techniques using the components and elements those frameworks provide.

    To give you an example, last week we were working in an app which was built using Ember.js framework, while we were in a manual code review activity, we found some duplicated code into three route components.

    So, in a nutshell beforeModel hook validates if an user has been logged in, and if not then it redirects to login page. There is obviously a duplicated code smell winking its eye, we need to move that code into “something" that allows us to reduce changes to this logic be replicated three times or many more if new routes are defined and need to be constraint by this logic.

    Remember previous post I mentioned code smells can be solved by using refactoring techniques? Move Method refactoring is the one I chose.

    Before I dig in it, a question emerges, where do I put that code i’m going to move?, here is where my framework understanding should be highlighted. Ember.js introduces concepts like Mixin and Services, which were my first options to use, but then I notice Ember uses routing hierarchy delegation and it is like implementing inheritance between routes. I took latter approach, the reason was because it was a routing problem and that piece of code should be also managed at the same level, there are not another components in the app that use that kind of logic so I discarded mixing and services. ApplicationRoute is the root parent of all routes so I applied move method refactoring to put my code into it. I started applying move method in one route, tested it and then move on the next one. Here is the result.

    The other routes don’t need to define this method anymore and it is also located in just one place.

    To conclude, there are a few things I want to highlight about this exercise.

    First thing is whatever framework, programming languages, even markup languages you use you are not except from introduce some code smells, this is something natural that happens most of time, the key is to learn to detected them and mitigate them using refactoring techniques.

    Second thing is you must have a basic understanding about the framework you use, get knowledge about overall components, what are their roles, many of frameworks were built following some architectural patterns so they can make your life easier, when applying a refactoring take in account framework components first, then see if it can be solved by using low level elements like classes, objects, etc.

    Finally, dedicate yourself at least 15 min of your day to learn about refactorings and code smells. For every code smell you learn try to find in your code if it exist, then apply the refactoring to mitigate it, this is the best way for training and put yourself into a different mindset. Practice and practice.

    Keep Learning, Keep Applying…

    Continue Reading
  • Larry Morales

    Code review: Refactoring (part I)

    Let’s imagine you want to visit a friend in town but you don’t know where he or she lives, so you call him/her and ask for his/her address. The address becomes your target place, it’s the place where you want to go. When you are about to decide to move, you think, what path should I take so it can lead me to my destiny?…So your options are probably based on less time path, or more secure path, or nearest path. After you decide to use a path, you move and follow it until you end at your destination (your friend’s house).

    That kind of thinking is a little bit related with the concept of refactoring. Imagine now that the situation is not to visit a friend but to extract a piece of code in your app. With that piece of code, you want it to live in a new method, the new method now becomes your target (the place where you want to go). So you think again, what path should I take to put that piece of code into a new method in a safe way?…

    Wait, safe way?, extract a piece of code can involve a lot of things, you could extract local variables which are used later in the method you extract your code from, code you extracted can also depend on some variables defined before from the code you extract, so this process should be done carefully as current behavior should be preserved.

    A refactoring is a prescription of a safe path you need to take in order to solve a particular problem in your code. I like to think about refactoring not as an activity where you remove duplicated code or make your code easier to understand in a deliberate way, a refactoring feels like a guideline, it is a step by step guide to transform your unhandled piece of code into a handled one in a safe way, without affecting code behaviour.

    Following the example I mentioned earlier, supposed that the code I want to extract is related to outstanding calculation:

    which path should I take? how should I extract that code? well, there is a refactoring called Extract method with the mechanics(path) to extract a piece of code into a method in a safe way. I use this refactoring almost every day and it has became pretty natural to me to apply. After applying Extract Method Refactoring my code looks like this:

    Look that code I extracted depends on customer variable. customer is just used to read some of its attributes (orders), so I passed it as a parameter to the function calculateOutstanding. However, outstanding variable is calculated and then it is used later in print details section inside printFor function, that’s the reason why it’s returned.

    Extract method is just one of many refactorings existing. Every refactoring has a template, if you click on Extract Method link mentioned before, you’ll see the first thing is a definition about that refactoring, and coming next you’ll see a before and after refactoring class diagrams, which means “go from this start point to this target point”, there is also a motivation to use the refactoring, mechanics(path or step by step) for applying it and some examples.

    Nature is full of balance and contrast…

    There is no existence of light if darkness would not exist, we would not talk about war if we don’t know what peace means, we inhale oxygen produced by plants to exhale carbon dioxide which feeds plants. So, we can not mention refactoring without talking about code smells. Refactoring would not exist if code smells do not emerge.

    In a simple way, code smells are those “things” that make you feel like there is something wrong in your code. When emerging, they can make your code hard to change or understand. Some of us recognised duplicated code introduced in an application. Duplicated code is considered as a code smell because of the impact it can cause when changes need to be done. You will need to introduce the changes as many times as the code is duplicated. Here is a list of some code smells you can probably deal with:

    Duplicated Code
    Long Method
    Large Class
    Long Parameter List
    Divergent Change
    Shotgun Surgery
    Feature Envy
    Data Clumps
    Primitive Obsession
    Switch Statements
    Parallel Inheritance Hierarchies
    Lazy Class
    Speculative Generality
    Temporary Field
    Message Chains
    Middle Man
    Inappropriate Intimacy
    Alternative Classes with Different Interfaces
    Incomplete Library Class
    Data Class
    Refused Bequest
    

    Code smells are out of scope for this post, I’ll talk about them in another entry, I think they are as important as refactoring. What I really want you to know is that refactorings help to mitigate a code smell, when a code smell is detected, there is at least one refactoring technique to make it disappear.

    Refactorings can be applied in an incremental approach. You start applying one small refactoring in a method, then a medium one in a class or object and after that you can apply a bigger one between two classes or components. This is an approach used in emergent design, the result for this process is to accomplish a more robust design which can suit to a pattern design. I recommend you take a look to Refactoring to Patterns by Josh Kerievsky.

    To conclude, refactoring (the verb) is an activity we as developers must consider to practice and learn more about it. Code smells are inherent attributes when developing software, they always seem to appear, learn to detect them and mitigate them using refactoring techniques require some practices and discipline, but the result is a code that looks cleaner, easy to change, easy to mantain and easy to extend.

    Keep learning, keep applying.

    Continue Reading
  • Larry Morales

    Colombia is getting mature and more competitive in software development

    In 2013, Organization for Economic Co-operation and Development accepted Colombia as a candidate to become an active member of this privileges countries group. If Colombia accomplishes all the requirements for membership requested by OECD, it would be part of “the best practices club”, where countries with better perspective and better performance in social and economic fields help each other to improve their public politics quality, which means more investment, more employment, more capability.

    There had been great progress in some areas along these two years. One of the greatest progress made in Colombia is related with IT investment. With help of ICT ministry and FITI, some programs were developed to help entrepreneurs to create businesses around IT technology, programs to capitalise IT professionals which offer free certifications, free scholarships for MBA and PHB degrees related to IT industry, I+D+I, just to name a few ones.

    An info-graphic emitted by ICT ministry for the last four months of 2014 shows how IT technology has improved its use and capability to coverage almost the whole country. Also, statistics from the same source show how GDP had behave in a decade thanks to IT investment.

    So things look good for Colombia in IT subject, but there’s a long road to follow and good quality software production has been the need nowadays.

    In Colombia software investment had grown up around 43% in the last decade, estimated in USD 6’200’000,000. Today, exists a lot of demands for software production in Colombia, many companies decides to offshore software production because of the lack of companies with good quality software production or because they are too expensive, So they decide to outsource with Indian or Chinese companies with some maturity in software production and maybe because they are cheaper ;).

    On the other side, the great news is that this situation is changing, we as local software producers have a big social responsibility on what we do and we’re doing a lot of improvement in quality of software production.

    Software Engineering Institute a.k.a SEI, emitted four months ago a report that keeps me motivated and dedicated to get better in this business (and also proud to be a colombian :), yeah I love my country). Colombia got the first place as the country with more software development companies which applies the best practices in software engineering in south america, according to the Capability and Maturity Model Integration a.k.a CMMI. This means Colombia is getting mature about the way it develops software and it’s making it more competitive in the business.

    This is a reality, Colombia is developing good quality software applying the best practices according to SEI. I hope the number of software development companies in Colombia with good quality software production keeps increasing as we follow a continuous improvement process, and maybe someday when Colombia becomes an active member of OECD would be prepared to be the reference in software development.

    Keep Learning, Keep applying…

    Continue Reading