DataMapper 2 Status and Roadmap

Posted by solnic on December 20, 2012 Leave a comment (20) Go to comments

We’ve been really busy working on the mapper part of DataMapper 2 for the last few months. This gives us more clarity about what is still missing. The mapper currently supports two different engines, Veritas and Arel. By introducing engines API we managed to create a nice database abstraction layer along with model and attribute mapping layers. The entire system works like a multi-level pipe so that we have 3 separate “levels” where data is being processed. Data fetched by a database driver is consumed by a gateway which sends data up to the mapping layer. The mapping part is pretty much done and powerful. We can map “raw” data sets to domain objects. We can map relationships by eagerly loading them from a database. We can map embedded values and collections too. This is pretty awesome already :) OK so what’s still missing?

Veritas “write” Support

Currently our veritas engine supports only reading from the databases. Dan Kubb will be working on finishing write support in Veritas and once that’s done we’ll be able to easily integrate it with dm-mapper. Veritas will also be renamed soon as people are confusing it with Virtus.

Arel Engine Using DataObjects

One of the things I want to tackle is to extract connection handling from ActiveRecord and port it to use DataObjects to achieve consistent behavior across all the drivers. Currently dm-mapper uses anonymous ActiveRecord::Base subclasses just to make it work with Arel. This was always a temporary solution and now it’s time to improve it. I believe this could become a win-win situation for both DataMapper and ActiveRecord projects if we could use the same drivers. I’m not sure if rails-core team would be interested in working on it though.

Coercions For Engines

I will soon extract Virtus::Coercion into a separate project and use it in dm-mapper so that engines could perform data coercions when a database driver doesn’t support it. This also requires a bigger addition to the dm-mapper where we will be able to define how certain ruby types are being coerced into values in a database. For example casting true object to “1″ or date time to time etc.

Session Integration

We have already a working persistence state-machine that now we need to integrate with dm-mapper. The concept is pretty simple. The mapper provides database access API and if you need things like “dirty tracking” or dependency resolution via UnitOfWork then you use a special session object that extends mapper API.

How is it going to look like?

Lots of things already work so here’s a short sample of how the API looks at the moment:

# In DataMapper 2 we don't pollute global constants with shared state.
# That's why we decided to use an environment object that *you*
# create and use do build mappers and configure everything.

# Let's call it "datamapper" :)

datamapper = DataMapper::Environment.new

# You use environment object to establish connection with a db
datamapper.setup :postgres, :uri => "postgres://localhost/test"

# So let's say we have 2 domain objects Page and Book

class Page
  include DataMapper::Model

  attribute :id, Integer
  attribute :content, String
end

class Book
  include DataMapper::Model

  attribute :isbn, String
  attribute :title, String
  attribute :author, String
  attribute :pages, Array[Page]
end

# You use environment object to build mappers

datamapper.build(Page, :postgres) do
  key(:id)
end

datamapper.build(Book, :postgres) do
  key(:isbn)

  # here we establish a relationship between page and books
  has 0..n, :pages, Page
end

# we need to finalize the env now
datamapper.finalize

# to access a mapper you use #[] method and model constant
# so to fetch all books with their pages you just do:

datamapper[Book].include(:pages).all

# Query API is pretty similar to what you already know

datamapper[Book].find(:author => 'John Doe').limit(10).offset(2)

# In DataMapper 2 we don't pollute global constants with shared state.

# That's why we decided to use an environment object that *you*

# create and use do build mappers and configure everything.

# Let's call it "datamapper" :)

datamapper = DataMapper::Environment.new

# You use environment object to establish connection with a db

datamapper.setup :postgres, :uri => "postgres://localhost/test"

# So let's say we have 2 domain objects Page and Book

class Page

include DataMapper::Model

attribute :id, Integer

attribute :content, String

end

class Book

include DataMapper::Model

attribute :isbn, String

attribute :title, String

attribute :author, String

attribute :pages, Array[Page]

end

# You use environment object to build mappers

datamapper.build(Page, :postgres) do

key(:id)

end

datamapper.build(Book, :postgres) do

key(:isbn)

# here we establish a relationship between page and books

has 0..n, :pages, Page

end

# we need to finalize the env now

datamapper.finalize

# to access a mapper you use #[] method and model constant

# so to fetch all books with their pages you just do:

datamapper[Book].include(:pages).all

# Query API is pretty similar to what you already know

datamapper[Book].find(:author => 'John Doe').limit(10).offset(2)

Keep in mind that by integrating DataMapper 2 with various frameworks we will simplify a lot of the things. We’ll be able to generate mappers for you, call finalize automatically etc.

Release Plan?

We want to push the first alpha as soon as we get all types of “primitive” coercions working. If I manage to port Arel to use DataObjects then this problem will be solved for rdbms databases and it’s going to be a good moment to push the first alpha. Maybe we’ll make it before end of the year ;P

After that we’ll continue with releasing next alpha versions until we’re certain that the foundation is solid and we can start the beta phase. Somewhere during that process Dan should be able to finish Veritas “write” support and we’ll start working on even more advanced things like cross-database interactions.

We need to go through lots of alpha/beta releases until we can tag 2.0.0 – after all it’s a completely new project.

Related Projects

DataMapper 2 is a pretty big ecosystem of various gems that we like extracting from core parts whenever we feel it makes sense. Here’s a list of all the extracted gems:

And here are the core parts we’re working on:

Share/Bookmark

Blogdatamapper, ruby

← Subclassing Module For Fun and Profit

Mutation testing with Mutant →

http://twitter.com/yanoo_ Jan Dudulski

Keep going, great work!
http://twitter.com/MostlyHarmlessD Dom Barker

Really looking forward to this all being finished!
http://twitter.com/lukeholder Luke Holder

thankyou for the update. Really want to use this in production but boss wants to wait until everything is 1.0 or \”completed\”. thanks for all your hard work.
http://twitter.com/ptico ptico

DataObjects is great. I really like it. But, do_postgres is ~1.5 times slower than pg driver and do_mysql ~1.1 times slower than mysql2 (i test only reading). Did you plan to do something with DO drivers performance in the nearest future?
- http://solnic.eu/ solnic
  
  Not sure about mysql but pg, as I wrote, doesn\’t do coercion so no wonder it\’s faster. This is IMHO an insignificant difference. If we find DO drivers to be too slow we can always work on making them faster.
- http://twitter.com/dkubb Dan Kubb
  
  One thing to keep in mind is that, as solnic notes, pg doesn\’t do any coercion of the data while DO does. That work has to happen somewhere along the way, it\’s just that DO coerces the value up-front, eliminating the intermediary object from being allocated. Any fair comparison to DO should include coercion if it\’s not already happening.
  
  The comparison to mysql2 is probably fair, provided that casting is turned on, which I think it is by default.
  
  Actually, I love the mysql2 interface that brianmario designed, and I\’ve spoken with dbussink (the DO maintainer) about it and he is open to the idea of simplifying the interface across all the drivers to be more similar. It should result in some performance benefits too.
  
  The main thing we\’d need to make this happen is help from the ruby community since we\’re all so focused on the mapper and query side of things; the DO drivers in their current state are good enough for our purposes at the moment, but performance improvements and simplifications definitely wouldn\’t be unwelcome ;)
Chris

What\’s wrong with being able to do this ? : Book.include(:pages).all
- http://solnic.eu/ solnic
  
  There\’s nothing \”wrong\” with this. It\’s just a different pattern. In Data Mapper domain objects have no knowledge about persistence. That\’s why the interface to access db is implemented in the mapping layer.
  
  You can read about Data Mapper pattern here: http://www.martinfowler.com/eaaCatalog/dataMapper.html
trans

As for shared Drivers, I suggested to DHH long ago that the driver portion of AciveRecord be split out into a separate project. He seemed in favor of the idea, but I did not purse it beyond an expiremental project. At the time I was thinking it would be called \”ActiveAdapter\” and I think DHH suggested \”AbstractAdapter\”. Anyway, the name doesn\’t matter. The point is that it was a good idea then, and still a good idea now.
DmitriyNesteryuk

I\’ve started using this cool project for interacting with our SOAP API. I would say it is amazing set of libraries, I learnt their code I was really impressed. The idea of abstracting from a persistent layer is a great. It can be used with any kind of storage (even API as in my case) From my experience it isn\’t difficult to work with your persistent layer through this library. The flexibility of those libraries allow you to do so. I am still looking forward to a release of this project, because some features haven\’t implemented yet, but as I said we have started using it and I am really glad that we had chosen this library.
http://www.facebook.com/saratovsource Александр Кириллов

DataMapper 2 is great gem! When you can integrate it with the rails? It will be a great gift for Christmas)) Thank you for your work!
hehejo

I’m very exited about the ongoing development of DataMapper 2 as I use DataMapper 1 as a beloved workhorse in my rails app.
I hope that DataMapper 2 fits well with Rails. :-)
Milovan Zogovic

Thank you for the great work! Put this on kickstarter, and community will greatly support this project.
http://yeban.in Anurag Priyam

Is it ok to expect that one would be able to:
1. re-open auto-generated mappers and make minor changes?
2. inherit (or mix-in) an auto-generated mappers to create new mappers?

Or is it just non-sensical to think about mappers that way?
- http://solnic.eu/ solnic
  
  There’s a mapper builder that you use to create mapper classes, it has a simple DSL to override the default behavior and settings. After all mappers are built and entire environment is set we freeze mapper classes so you won’t be able to re-open them later.
  - http://yeban.in Anurag Priyam
    
    Ok, right. It doesn’t make sense to think about re-opening mappers once frozen. But the DSL would help if I want to override defaults, which is what I meant by minor changes.
    
    Now say I define a mapper class using the mapper builder. Can I then inherit from it like any other Ruby class and wrap mapper methods to do something extra? For example, to cache the results to memcache maybe. Something like:
    
    https://speakerdeck.com/solnic/wroc_loverb-beyond-the-orm-an-introduction-to-datamapper-2?slide=49
    
    class User class CachedMapper < Mapper def find(*args) cache(super(*args)) end end end
    
    Thanks, @solnic.
    - http://solnic.eu/ solnic
      
      Yes those are normal Ruby classes :) You can do whatever you want with them except changing them.
      - http://yeban.in Anurag Priyam
        
        Awesome! “Normal” Ruby classes is what I was looking for. So things should work as long as I don’t change the method contract. Sometimes stuff that is held together by too much magic breaks when you try to do just a little bit more with it.
        
        Thanks a lot, @solnic:disqus, for taking the time to answer my questions. And for developing DM2: I am super excited about DM2 and the bunch of libraries you all are creating in the process. I have a few more questions that I will post in a separate thread. I hope you will find the time to answer them too :).
      - http://solnic.eu/ solnic
        
        One of our motivations behind the API is to have “magic” with easy to use DSLs but at the same time have lower level API’s (which are still public) that you can use too if you want and need full power. We’re as much focused on “the guts” of the system as on the user-facing parts so that it’s clean and easy to use on all the levels. So for example you should be able to create a mapper instance with a one-liner but you should also be able to build a custom mapper class “manually” and adjust various settings yourself and instantiate it yourself.
      - http://yeban.in Anurag Priyam
        
        You probably wanted to make sure that I (or someone reading this thread) don’t take “Normal” Ruby classes to mean DM2 is “magic” free. I think, having clean, lower level, manual APIs and then using Reflection to create a less-work / standard-use-case interfaces on top is a great way to do things.

solnic.eu

probably the best tagline in the world