JSON validation and error handling in Play 2.1

Posted by on April 21, 2013 Comments Off

Play framework 2.1 comes with a slightly new approach to reading JSON values. In version 2.0, there was a Reads[T] trait with a single method reads(json: JsValue): T, which simply took the JSON object and converted it to the domain object. With such a simple approach, it was very difficult to handle various conversion errors such as missing field, incorrect data types, etc. Since it was not even possible to return something like Either[ErrorDescription, T] from the reads method, it was necessary to throw and handle various parsing exceptions. With the new Play framework 2.1, thankfully, this has changed.

Play 2.1 reads method doesn’t return T directly anymore, but rather a JsResult[T], which can be either a JsSuccess[T] if the read succeeded or JsError[T] otherwise. Examples and deeper explanation can be found in the documentation. I would like to present here, how you can utilize the JsError object to automatically notify the clients about conversion/validation errors.

Consider an example scenario, when we are writing a RESTful web service, which accepts JSON objects representing user registrations. We can represent the user by the following case class:

case class User(id: Option[Long], email: String, 
    password: String, fullname: Option[String])

Let’s create an implementation of the Format[User] trait, which converts between the domain object and its JSON representation:

implicit object UserJsonConverter extends Format[User] {
  def reads(json: JsValue): JsResult[User] = (
    (__ \ "id").readNullable[Long] and
      (__ \ "email").read(email) and
      (__ \ "password").read(minLength[String](8)) and
      (__ \ "fullname").readNullable[String])(User.apply _).reads(json)
 
  def writes(o: User): JsValue = Json.obj(
    "id" -> o.id,
    "email" -> o.email,
    "password" -> o.password,
    "fullname" -> o.fullname
  )
}

Notice how we handle not just the JSON conversion here, but also some validation logic, such as minimal password length or correct e-mail format. You might see some resemblance with parser combinators I wrote about in the last post; you might also write the and combinator as ~ to emphasise they are both monadic combinators.

In the action handler, we can now parse the JSON contained in the body of the POST request:

def post = Action {
  request =>
    request.body.asJson.map(_.validate[User] match {
      case JsSuccess(user, _) => // store the user in the database
      case err@JsError(_) => BadRequest(Json.toJson(err))
    }).getOrElse(BadRequest)
}

Here, we validate the JSON as the User instance and pattern match whether or not it succeeded. If so, we can obtain the User instance from the JsSuccess and pass it to a further processing. If the result is JsError, however, we need to somehow inform the user that the conversion/validation failed. If we are writing a RESTful web service, for example, it might be handy to return a JSON containing some information about the validation errors. We can write an implicit Writes[JsError], which would serialize the JsError object into JSON. Here is one of the possible implementations:

implicit object JsErrorJsonWriter extends Writes[JsError] {
  def writes(o: JsError): JsValue = Json.obj(
    "errors" -> JsArray(
      o.errors.map {
        case (path, validationErrors) => Json.obj(
          "path" -> Json.toJson(path.toString()),
          "validationErrors" -> JsArray(validationErrors.map(validationError => Json.obj(
            "message" -> JsString(validationError.message),
            "args" -> JsArray(validationError.args.map(_ match {
              case x: Int => JsNumber(x)
              case x => JsString(x.toString)
            }))
          )))
        )
      }
    )
  )
}

So, if we post an empty JSON – {} – to the post action, for example, we would receive the following JSON with the description of validation errors:

{
  "errors": [
    {
      "validationErrors": [
        {
          "args": [], 
          "message": "validate.error.missing-path"
        }
      ], 
      "path": "/email"
    }, 
    {
      "validationErrors": [
        {
          "args": [], 
          "message": "validate.error.missing-path"
        }
      ], 
      "path": "/password"
    }
  ]
}

What is beautiful about this approach is that you get conversion/validation problem notification almost for free. You only have to write a Reads[JsError] object (or use the one I presented), make it implicit in the scope and you are all set. Then, you can run validate on the JSON object you receive and in case of an error, return the serialized JsError instance to the client. This can be particularly handy when you are writing a web service and communicate with the client with JSON objects only. If the action handler, on the other hand, generates the view directly, the standard Play form validation approach is a probably a better option.

Parser combinators (and what if regular expressions are not enough)

Posted by on June 11, 2012 Comments Off

Regular expressions are the most widely used technique of describing text. All languages somehow support regular expressions, they are relatively easy to use and all over the internet there are predefined ones for many common scenarios (such as for a correct e-mail, domain name, etc.). They might look, therefore, as an universal solution for all text manipulation needs. In this article, I would like to show there is an other, more powerful, way of processing text which is as easy to use as regexps – parser combinators.

I first encountered parser combinators during Haskell courses on the university. Unfortunately, my attitude that time was something like “Don’t bother me with monads, I wanna do real-world. Give me Spring and Hibernate!” However, now, when doing real-world, I see how invaluable these theoretical foundations are and how often I can benefit from it in practice.

The idea behind parser combinators is quite simple, two fundamental elements are parsers and parser combinators. Parsers recognize and process some input. They can either succeed or fail. Parser combinators, on the other hand, combine existing parsers into more complex ones. Therefore, complicated parsers can be built from bottom up, using an appropriate composition of simpler parsers, which are easier to understand, implement and test. Let’s take a look at several scenarios where parser combinators are used as a convenient alternative to regular expressions (or rather their extension – as you will see later). Code examples are written in Scala, which provides very nice DSL for this purpose. However, parser combinator libraries exist for many languages and once you understand the concept, it shouldn’t be difficult to port it across different languages.

Simple example: parsing IPv4 addresses

Let’s take a look at the following code snippet:

def parseUsingRegexp(input: String): Option[(Int, Int, Int, Int)] = {
  val IP = """b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)b""".r 
  input match {
    case IP(a, b, c, d) => Some(a.toInt, b.toInt, c.toInt, d.toInt)
    case _ => None
  }
}

Unless you are Perl hacker, it took you a couple of seconds to find out what is actually going on in here. It is a method that receives a string on the input and returns either None if the string is not correct IPv4 address or a tuple of 4 integers representing the four parts of the IPv4 address. The regular expression on the second line is copied from a website collecting useful regexps. As we can see, such expression is very complicated (163 characters long) and thus error-prone (just imagine you accidentally delete one character – nobody might ever find out). Moreover, it is also repetitive. Since IPv4 contains four 1-byte numbers separated with dots, it would be great if we could somehow define the pattern for one group and repeat it several times. Indeed, we might have used quantifiers to specify exact repetition, but then we wouldn’t have had access to captured groups.

Now let’s take a look, how the same functionality can be achieved using parser combinators:

def parseUsingParserCombinator(input: String): Option[(Int, Int, Int, Int)] = {
  def dot: Parser[Any] = "."
  def ipGroup: Parser[Int] = "25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?".r ^^ (_.toInt)
  def ip: Parser[(Int, Int, Int, Int)] = (ipGroup <~ dot) ~ (ipGroup <~ dot) ~ (ipGroup <~ dot) ~ ipGroup ^^ { 
    case (a ~ b ~ c ~ d) => (a, b, c, d)
  }
  parseAll(ip, input) match {
    case Success(r, _) => Some(r)
    case _ => None
  }
}

Here we have 2 simple parser combinators: dot, simply denoting a dot, and ipGroup for numbers 0-255. The third parser, ip is created as a sequential composition (parser combinator ~) of those two simpler ones. Sequential composition means that the parsers chained with ~ will be executed consecutively, one after another, and the computation succeeds if all the parsers in the chain succeed. There is also parser combinator <~, which is doing the same as ~, it just forgets the result of its right argument, returns only the result of the left one and thus passes it to a further processing (here we are not interested in any further processing of dots).

All parsers have a type. The type of a parser denotes what does it produce when it successfully reads the input. For example, the type of the parser ipGroup is Parser[Int], which means that after the parser reads its portion of the input, it will result in an Int. In order to convert parser results, there is the parser combinator ^^, which takes a transformation function as an argument.

We can now ask what can actually be a parser. In case of the dot parser, we assign an ordinary string, while ipGroup seems to be a regular expression. In reality, all the parsers are obliged to extend (Input => ParseResult[T]), but we can use strings and regular expressions as parsers because they are implicitly converted to Parser[String]. However, this is a Scala-specific issue, not applicable in other languages. For details consult its documentation.

School example: Infix to prefix conversion

Let’s go to another example, now something what cannot be done using regular expressions. We have a following grammar producing boolean expressions:

Expr ::= Elem and Elem | Elem or Elem | Elem
Elem ::= not Token | Token
Token ::= true | false | (Expr)

Examples of strings generated by this grammar are true, false, not true, not (true and false), true or (not false), etc. We can see that this language is not regular anymore, yet context-free. The idea of the proof is very simple – just imagine an n-state finite automaton and an expression starting with n+1 opening parentheses. How could such machine verify that all opening parentheses are properly closed? Therefore, the language generated by such grammar cannot be described by regular expressions.

Let’s say we want to write a method which would check if an expression passed as an argument is correct and convert it into the Scheme-like prefix notation (incl. converting values true and false to #t and #f, respectively). So, for example true or false becomes (or #t #f), true and not (false or true) becomes (and #t (not (or #f #t))), etc. I remember doing something similar few years back as a home assignment in C. If I had known parser combinators that time, the number of lines necessary would have reduced to one fifth or something like that.

Our strategy will be following: instead of directly manipulate the input string, we will first convert the expression into our internal representation and then create a new string representing the very same expression, just in the prefix notation. Our object model will look like this:

sealed trait Expr
case class Atom(value: Boolean) extends Expr
case class Not(expr: Expr) extends Expr
case class And(left: Expr, right: Expr) extends Expr
case class Or(left: Expr, right: Expr) extends Expr
case class Paren(value: Expr) extends Expr

We will need to write three parsers in total. Each one will correspond to a specific rule of the grammar:

def expr: Parser[Expr] =
    (elem <~ whiteSpace ~ "and" ~ whiteSpace) ~ elem ^^ {case x ~ y => And(x, y)} |
    (elem <~ whiteSpace ~ "or" ~ whiteSpace) ~ elem ^^ {case x ~ y => Or(x, y)} |
    elem
 
def elem: Parser[Expr] = "not" ~> whiteSpace ~> atom ^^ {Not(_)} | atom
 
def atom: Parser[Expr] = ("true" | "false") ^^ {x => Atom(value = x.toBoolean)} | 
    "(" ~> expr <~ ")"

Now, when our parsing infrastructure is ready, we only need to write a method which triggers parsing and then converts our internal representation of the expression into the prefix notation:

def convert(input: String) = {
  def doConvertToInfix(expr: Expr): String = expr match {
    case x: And =>
      "(and " + doConvertToInfix(x.left) + " " + doConvertToInfix(x.right) + ")"
    case x: Or =>
      "(or " + doConvertToInfix(x.left) + " " + doConvertToInfix(x.right) + ")"
    case x: Not =>
      "(not " + doConvertToInfix(x.expr) + ")"
    case x: Atom => if (x.value) "#t" else "#f"
    case x: Paren => doConvertToInfix(x.value)
  }
 
  def parseExpression(s: String): Expr = parseAll(expr, s) match {
    case Success(res, _) => res
    case Failure(msg, _) => throw new Exception(msg)
  }
 
  val ast = parseExpression(input)
  doConvertToInfix(ast)
}

Real-world example: parsing chess board state

In chess programming, there are six elements necessary to remember in order to reconstruct the game at the very same state: the position of pieces on the board, the player whose turn it currently is, available castlings, en passant square, the number of halfmoves since the last capture and the current move number. There is a de-facto standard notation for this purpose, called Forsyth–Edwards Notation (FEN). For details of the structure of games stored in this format, please consult the wiki, its documentation or F.A.Q. FEN string of a chess game in the default position is following: rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1.

Parsing a FEN string and converting it to some internal structure representing a game state is not that difficult. Without the use of some more advanced techniques, however, it would require complicated nested loops and long switches. I will present here, how parsing such string using parser combinators can be elegant, concise and fun.

Let’s have a following object model:

// Piece representation
sealed abstract class Piece {
  val white: Boolean
}
 
object Piece {
  def apply(str: String): Piece = str match {
    case "p" => new Pawn(false)
    case "P" => new Pawn
    case "r" => new Rook(false)
    case "R" => new Rook
    ...
    case _ => 
      throw new IllegalArgumentException("Passed incorrect piece representation")
  }
}
 
case class Pawn(white: Boolean = true) extends Piece
case class Rook(white: Boolean = true) extends Piece
case class Knight(white: Boolean = true) extends Piece
case class Bishop(white: Boolean = true) extends Piece
case class Queen(white: Boolean = true) extends Piece
case class King(white: Boolean = true) extends Piece
 
// Castling representation
sealed abstract class Castling {
  val white: Boolean
}
 
object Castling {
  def apply(str: String): Castling = str match {
    case "k" => new KingsideCastling(false)
    case "K" => new KingsideCastling
    case "q" => new QueensideCastling(false)
    case "Q" => new QueensideCastling
    case _ => 
      throw new IllegalArgumentException("Passed incorrect castling representation")
  }
}
 
case class KingsideCastling(white: Boolean = true) extends Castling
 
case class QueensideCastling(white: Boolean = true) extends Castling
 
 
// Game representation
class GameState private(val board: List[Option[Piece]], val whiteOnTurn: Boolean, 
                        val castlingAvailability: Set[Castling],
                        val enPassantSquare: Option[Int], val halfMoveClock: Int, 
                        val fullmoveNumber: Int)
 
 
object GameState {
  def apply(fen: String): Option[GameState] = ... // parsing will be here
}

If we implement GameState this way, we will be able to use the following very concise code to convert a string, containing a game in FEN, into our internal representation:

val fen = "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"
val gameState = GameState(fen)

Let’s now take a look what actually do we need to parse. The FEN string consist of six elements separated with whitespace. Therefore, our parser will be a sequential composition of eleven parsers, six ones parsing individual elements of the FEN string and five ones parsing whitespace between elements. Let’s start with the most simple ones. Last two elements are simple integers. Our parser, therefore, will be simple regular expression matching integers:

val intParser: Parser[Int] = """d+""".r ^^ (_.toInt)

Fourth element denotes en passant square. It is written as a square in algebraic notation (e.g. a1, c6, h8, …). However, in the GameState class, we store all fields as integers, starting with 0 up to 63, hence our parser will be of type Parser[Option[Int]] and not Parser[String]. Option is used because the en passant square might be empty (and thus written as - in FEN).

val enpassantSquareParser: Parser[Option[Int]] = "-" ^^ (_ => None) | 
    ("a" | "b" | "c" | "d" | "e" | "f" | "g" | "h") ~ 
    ("1" | "2" | "3" | "4" | "5" | "6" | "7" | "8") ^^ 
    {case x ~ y => Some(8 * (y.head - '1') + (x.head - 'a'))}

Third element in a FEN string represents available castlings. Since it does not make sense to hold duplicate values, available castlings are represented as a set in our object model. The parser, therefore, is of type Parser[Set[Castling]]:

val castlingParser: Parser[Set[Castling]] = 
    "-" ^^ {_ => Set[Castling]()} | 
    rep("k" | "K" | "q" | "Q") ^^ {_.map {x => Castling(x)}.toSet}

The current player’s color is denoted by the second element in FEN. It is either “w” or “b”. In the object model, the color is denoted by a boolean value passed to the constructor of all pieces. Hence the type of the parser is Parser[Boolean]:

val activeColorParser: Parser[Boolean] = ("w" | "b") ^^ {_ == "w"}

Not let’s get to the most complicated parser which is responsible for parsing the first element of the FEN string – the position of pieces on the board. In our object model, pieces on the board are represented by a list of exactly 64 elements of type Option[Piece]. It means that if there is a piece p on a square i of a board board, then board(i) = Some(p); or board(i) = None otherwise. Therefore, the type of our parser will be Parser[List[Option[Piece]]].

When we look at the format of the first FEN element, we can see some repetition there. The string consists of 8 strings describing position of pieces on a particular rank (= row on the chessboard), separated with a slash. We can thus create a parser which parses one rank and parsing of the whole board will be a composition of rank parsers. We can proceed even further – the rank parser parses an input which consists either of characters representing the pieces on the board, or of digits denoting how many consequent empty squares there are. Let’s create separate parsers for them, too:

val occupiedSquaresParser: Parser[List[Option[Piece]]] = 
    ("p" | "P" | "r" | "R" | "n" | "N" | "b" | "B" | "q" | "Q" | "k" | "K") ^^ { 
      x => List(Some(Piece(x)))
}
 
val emptySquaresParser: Parser[List[Option[Piece]]] = 
    ("1" | "2" | "3" | "4" | "5" | "6" | "7" | "8") ^^ {
      n => List.fill(n.toInt)(None)
}

Now, when we are able to parse individual characters, let’s create the rank parser:

val rankParser: Parser[List[Option[Piece]]] = 
    rep(occupiedSquaresParser | emptySquaresParser) ^^ {_.flatMap {x => x}}

You can see that the parser combinator rep is used here. It applies specified parser as many times as possible and its result is of type List[A], where A is the type of the parser passed as an argument. In this case, both occupiedSquaresParser and emptySquaresParser are of type Parser[List[Option[Piece]]] and thus the result of the parser rep would be List[List[Option[Piece]]]]. Therefore, it is necessary to merge all the nested lists into a single one using flatMap().

Now, we can use our rankParser to create the parser of the whole board:

val boardParser: Parser[List[Option[Piece]]] = repsep(rankParser, "/") ^^
        (_.reverse.flatMap(x => x))

You might wonder, why the list is reversed in the boardParser. The FEN string describes the board starting from the black’s side (back rank), but in our representation, we start from white’s perspective.

All necessary helper parsers are ready, let’s now create a complete parser, converting a string in FEN notation into our object model:

val fenParser: Parser[GameState] =                                           
    (boardParser <~ whiteSpace) ~ (activeColorParser <~ whiteSpace) ~
    (castlingParser <~ whiteSpace) ~ (enpassantSquareParser <~ whiteSpace) ~ 
    (intParser <~ whiteSpace) ~ intParser ^^
    {case a ~ b ~ c ~ d ~ e ~ f => new GameState(a, b, c, d, e, f)}

Do you see the beauty here? Such complicated parsing logic written as a combination of simple, easily understandable tasks.

The last necessary step is to write a method which triggers parsing itself:

def doParse(fen: String): Option[GameState] = parseAll(fenParser, fen) match {
  case Success(result, _) => Some(result)
  case Failure(msg, _) => throw new Exception(msg)
}

Conclusion

In this article I’ve shown how parser combinators work, how can we use them as a complement to regular expressions and an example where parser combinators are superior to regexps in parsing a language, which is not regular. You can find complete sources to all the examples from this article, as well as various ScalaTest tests, at my Bitbucket repository.

What makes programming languages really powerful?

Posted by on May 9, 2012 5 comments

It is quite common to hear discussions in programmers’ circles about how wonderful programming language X is because it has so cool features Y1, Y2, …, Yn and how unfortunate the programmers in Z (Z != X) must be because their language doesn’t have them. I always do my best not to get involved because I don’t find such discussions productive. Simply everyone should use tools which suit the needs best. However, I would like to have a think about what actually makes programming languages powerful and what only gives an impression of power.

While looking for a reference documentation of some string-manipulation methods in Python, I found out on a certain website that: “One of Python’s coolest features is the string format operator %”. I stopped for while and thought about how can something like that be one of the best features of a programming language. I perfectly understand what the author meant; it cannot be taken literally, he just meant it is very convenient to format a string using this syntax (and it certainly is). It kind of reflects though, how people often think about languages and their features; what the languages can and what cannot do.

In an ideal world, the programming language should not provide native language support for formatting a string, but rather provide a support for writing abstractions which resemble native language support. For example in Scala, there is no built-in string-formatting operator, only the format() method defined for strings that is doing more or less the same thing as % in python. It’s very easy, however, to enrich the String class with % method which, when written in infix operator notation, acts exactly like its Python cousin:

class PythonizedString(val str: String) {
  def %(args: Any*) = str.format(args: _*)
}
 
implicit def stringToPythonizedString(s: String) = new PythonizedString(s)
 
 
// And from now on, we can use Python-like syntax for string formatting
val greeting = "%s, %s!" % ("Hello", "World")
println(greeting) // => "Hello, World!"

This solution doesn’t mess the language syntax because it’s not a native construct, but rather an ordinary method defined (through an implicit conversion to a wrapper class) for all string objects and can be easily dropped by simply not importing the implicit conversion. Moreover, it is much more flexible because in the wrapper class PythonizedString, we can do whatever we can think of (call the existing formatting function format(), implement arbitrary string formatting logic, call a web service, etc.). In Python, however, as far as I know, we are stuck with the default implementation of the % operator.

Another example might be automatic resource management introduced in Java 7. Such feature needs a native language support only because Java does not (yet) support closures. The following snippet shows a possible approach to the automatic resource closing in Scala using a high-order function:

def use[A <: Closeable, B](stream: A)(closure: A => B): B = {
  try {
    val result = closure(stream)
    result
  } finally {
    if (stream != null) stream.close()
  }
}
 
val firstLine = use(new BufferedReader(new FileReader(("/home/semberal/scala/Sample.scala")))) {
  _.readLine()
}
 
println(firstLine)

I believe that new features should be added into the language very carefully, with strong emphasis on its impact on the flexibility of the language. Introducing a new special syntax only because of a stream-closing feature is, in my opinion, unfortunate. On the other hand, adding closures, which would allow creating library abstractions (including those, which would automatically close streams), would be a huge step forward. The problem is that now, when the automatic resource management feature is already part of the language, it will never go away; even though it won’t be necessary anymore once closures are added to the language (as currently planned for Java 8).

No additional flexibility for a high cost of a polluting the source code with constructs which would be never possible to drop from the language is also a reason why I think that adding XML literals to Scala was not a good design decision; however I might understand the intentions of Scala authors to provide a native language support for such a popular format. Not only it makes source code look ugly with embedded snippets of xml, while not adding any additional flexibility, but also raises questions like: “Should we add a native language support for JSON when various JavaScript frameworks or NoSQL databases using it are now so popular? Or what about any other data formats?” Language designers cannot predict which formats will arise in the future and, therefore, languages should ideally provide features to create abstractions allowing to work with any format in a reasonable way.

So, to sum up my ideal world, I don’t want a language which has a powerful feature Y; I want a language which gives me powerful built-in language constructs to create Y’, the syntax of which would be difficult to distinguish from the syntax of a built-in language feature.

Note: I’m aware that this blog post might resemble the unproductive discussion described in the first paragraph, but my point is more about the idea of extensibility of programming languages in general, not the particular languages themselves.

Update: There is a follow-up discussion at this article’s reddit page.