Package uk.ac.warwick.dcs.sherlock.module.model.base.preprocessing

Class StandardTokeniser

java.lang.Object

uk.ac.warwick.dcs.sherlock.module.model.base.preprocessing.StandardTokeniser

All Implemented Interfaces:

ITokenStringifier

public class StandardTokeniser
extends java.lang.Object
implements ITokenStringifier

Constructor Summary

Constructors

Constructor	Description
StandardTokeniser()

Method Summary

Modifier and Type	Method	Description
static int	preserveCommentLines(java.util.List<IndexedString> output, java.lang.StringBuilder active, int lineCount, org.antlr.v4.runtime.Token t, org.antlr.v4.runtime.Vocabulary vocab)
java.util.List<IndexedString>	processTokens(java.util.List<? extends org.antlr.v4.runtime.Token> tokens, org.antlr.v4.runtime.Vocabulary vocab)	Tokenises a file in the form of a list of tokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StandardTokeniser

public StandardTokeniser()

Method Detail

processTokens

public java.util.List<IndexedString> processTokens(java.util.List<? extends org.antlr.v4.runtime.Token> tokens,
                                                   org.antlr.v4.runtime.Vocabulary vocab)

Tokenises a file in the form of a list of tokens

Specified by:

processTokens in interface ITokenStringifier

Parameters:

tokens - the file as a list of tokens

vocab - the lexer vocab

Returns:

indexed lines of the file, tokenised

preserveCommentLines

public static int preserveCommentLines(java.util.List<IndexedString> output,
                                       java.lang.StringBuilder active,
                                       int lineCount,
                                       org.antlr.v4.runtime.Token t,
                                       org.antlr.v4.runtime.Vocabulary vocab)