WEEK TWO MS-1

After having issues in building cassandra_v2 ans server in general  on OS X,
I switched over development from OS X to Linux by installing a fresh copy of CentOS7.
I had MariaDB server build on it successfully and cassandra_v2 skeleton incantation as well.
The changes were pushed to github and the commits are found here

WEEK ONE MS-1

This week, I started working on my GSoC project. First, I did findings on how CQL differs from SQL and what query clauses work in MariaDB and not in Cassandra.
CQL is purposefully similar to SQL used in relational databases like MariaDB and MySQL. This similarity lowers the barrier of entry for users familiar with relational databases. Many queries are very similar between the two. In fact, a lot of basic things are even exactly the same. The following is both valid CQL and SQL

Switching databases

USE testatebase;

Creating Tables

CREATE TABLE IF NOT EXISTS testTable (id INT PRIMARY KEY);

Altering Tables

ALTER TABLE testTable ADD newField INT;

Creating Indexes

CREATE INDEX myIndex ON testTable (newField);

Inserting Data

INSERT    INTO    testTable    (id, newField) VALUES (1, 7);

Selecting Data

SELECT * FROM testTable WHERE newField = 7;

Counting Data

SELECT COUNT(*) FROM myTable;

Deleting Data

DELETE FROM testTable WHERE newField = 7;

 

Differences

Cassandra is a non-relational database, and so uses different concepts to store and retrieve data. In simple terms, a Cassandra keyspace is a SQL database, and a Cassandra column family is a SQL table (CQL allows you to interchange the words “TABLE” and “COLUMNFAMILY” for convenience). This difference necessitates a different syntax for creating and manipulating data:
-Creating databases:

/* Create a new keyspace in CQL */
CREATE KEYSPACE myDatabase WITH replication = {‘class’: ‘SimpleStrategy’, ‘replication_factor’: 1};

/* Create a new database in SQL */
CREATE DATABASE myDatabase;

Cassandra’s keyspaces require more specifications than a standard relational database. Note that the example above is the simplest form. In a production environment, likely spread across several data-centers, a keyspace would be created with a different strategy and replication factor.

  -Organizing Data

None of the standard relational stuff is going to work in CQL. There is no support for things like JOIN, GROUP BY, or FOREIGN KEY. Leaving these features out is important because it makes writing and retrieving data from Cassandra much more efficient.But sets of data tend to have relationships with one another. So without the relational tools of SQL, we need another strategy to represent these relationships. The problem is that reads in Cassandra tend to be more expensive than you might be used to when dealing with a relational database. Also with CQL, any query you plan to do in the future should already be organized into a column family. You want to look up addresses that correspond to a person? Then make a column family that consists of people and addresses. You can still store them separately, but you should also store them together. Having a column family to represent every query makes reads much more efficient. You’re not just going to have to de-normalize, you’re going to have to forget about normalization all together.

-Inserts vs Updates

The concept of cheap writes extends to updating data. This is because, unlike in SQL, a read is not performed during the update. The syntax is the same in CQL and SQL.

/* Updating data */
UPDATE    testTable    SET    newField = 2    WHERE   id  =  6;

However, if the row does not exist, it will still get created. Similarly as unintuitive, an INSERT statement will actually replace data if it exists. Because again, CQL does not perform a read while inserting. Without a read, there is no way to know if the data being inserted is replacing an existing record. This means that both inserts and updates are extremely fast.

-Time to Live

CQL enables you to set a TTL on a row. Meaning that you can set a row to expire 24 hours from the time it gets created. This is accomplished with the USING TTL command (values are in seconds).

/* Expiring Data in 24 Hours */
INSERT    INTO    testTable (id, newField)    VALUES (2, 9)     USING    TTL   86400;

Twenty-four hours after that query is executed, the data will be deleted.

These are some of the differences between CQL and SQL and cleary CQL is something less powerful and lack most complex operations that can be performed with SQL.


	

Community Bonding

During this period. I forked  and cloned the Server code. This can be found on my github repository here.

I also studied the storage engine definition class.

/** @brief
Class definition for the storage engine
*/
class ha_cassandra: public handler
{
friend class Column_name_enumerator_impl;
THR_LOCK_DATA lock; ///< MySQL lock
CASSANDRA_SHARE *share; ///< Shared lock info

Cassandra_se_interface *se;

/* description of static part of the table definition */
ColumnDataConverter **field_converters;
uint n_field_converters;

CASSANDRA_TYPE_DEF *default_type_def;
/* description of dynamic columns part */
CASSANDRA_TYPE_DEF *special_type_field_converters;
LEX_STRING *special_type_field_names;
uint n_special_type_fields;
DYNAMIC_ARRAY dynamic_values, dynamic_names;
DYNAMIC_STRING dynamic_rec;

ColumnDataConverter *rowkey_converter;

bool setup_field_converters(Field **field, uint n_fields);
void free_field_converters();

int read_cassandra_columns(bool unpack_pk);
int check_table_options(struct ha_table_option_struct* options);

bool doing_insert_batch;
ha_rows insert_rows_batched;

uint dyncol_field;
bool dyncol_set;

/* Used to produce ‘wrong column %s at row %lu’ warnings */
ha_rows insert_lineno;
void print_conversion_error(const char *field_name,
char *cass_value, int cass_value_len);
int connect_and_check_options(TABLE *table_arg);
public:
ha_cassandra(handlerton *hton, TABLE_SHARE *table_arg);
~ha_cassandra()
{
free_field_converters();
delete se;
}

/** @brief
The name that will be used for display purposes.
*/
const char *table_type() const { return “CASSANDRA”; }

/** @brief
The name of the index type that will be used for display.
Don’t implement this method unless you really have indexes.
*/
const char *index_type(uint inx) { return “HASH”; }

/** @brief
This is a list of flags that indicate what functionality the storage engine
implements. The current table flags are documented in handler.h
*/
ulonglong table_flags() const
{
return HA_BINLOG_STMT_CAPABLE |
HA_REC_NOT_IN_SEQ |
HA_NO_TRANSACTIONS |
HA_REQUIRE_PRIMARY_KEY |
HA_PRIMARY_KEY_IN_READ_INDEX |
HA_PRIMARY_KEY_REQUIRED_FOR_POSITION |
HA_NO_AUTO_INCREMENT |
HA_TABLE_SCAN_ON_INDEX;
}

/** @brief
This is a bitmap of flags that indicates how the storage engine
implements indexes. The current index flags are documented in
handler.h. If you do not implement indexes, just return zero here.
@details
part is the key part to check. First key part is 0.
If all_parts is set, MySQL wants to know the flags for the combined
index, up to and including ‘part’.
*/
ulong index_flags(uint inx, uint part, bool all_parts) const
{
return 0;
}

/** @brief
unireg.cc will call max_supported_record_length(), max_supported_keys(),
max_supported_key_parts(), uint max_supported_key_length()
to make sure that the storage engine can handle the data it is about to
send. Return *real* limits of your storage engine here; MySQL will do
min(your_limits, MySQL_limits) automatically.
*/
uint max_supported_record_length() const { return HA_MAX_REC_LENGTH; }

/* Support only one Primary Key, for now */
uint max_supported_keys() const { return 1; }
uint max_supported_key_parts() const { return 1; }

/** @brief
unireg.cc will call this to make sure that the storage engine can handle
the data it is about to send. Return *real* limits of your storage engine
here; MySQL will do min(your_limits, MySQL_limits) automatically.
@details
There is no need to implement …_key_… methods if your engine doesn’t
support indexes.
*/
uint max_supported_key_length() const { return 16*1024; /* just to return something*/ }

int index_init(uint idx, bool sorted);

int index_read_map(uchar * buf, const uchar * key,
key_part_map keypart_map,
enum ha_rkey_function find_flag);

/** @brief
Called in test_quick_select to determine if indexes should be used.
*/
virtual double scan_time() { return (double) (stats.records+stats.deleted) / 20.0+10; }

/** @brief
This method will never be called if you do not implement indexes.
*/
virtual double read_time(uint, uint, ha_rows rows)
{ return (double) rows / 20.0+1; }

virtual void start_bulk_insert(ha_rows rows, uint flags);
virtual int end_bulk_insert();

virtual int reset();
int multi_range_read_init(RANGE_SEQ_IF *seq, void *seq_init_param,
uint n_ranges, uint mode, HANDLER_BUFFER *buf);
int multi_range_read_next(range_id_t *range_info);
ha_rows multi_range_read_info_const(uint keyno, RANGE_SEQ_IF *seq,
void *seq_init_param,
uint n_ranges, uint *bufsz,
uint *flags, Cost_estimate *cost);
ha_rows multi_range_read_info(uint keyno, uint n_ranges, uint keys,
uint key_parts, uint *bufsz,
uint *flags, Cost_estimate *cost);
int multi_range_read_explain_info(uint mrr_mode, char *str, size_t size);

private:
bool source_exhausted;
bool mrr_start_read();
int check_field_options(Field **fields);
int read_dyncol(uint *count,
DYNAMIC_COLUMN_VALUE **vals, LEX_STRING **names,
String *valcol);
int write_dynamic_row(uint count,
DYNAMIC_COLUMN_VALUE *vals,
LEX_STRING *names);
void static free_dynamic_row(DYNAMIC_COLUMN_VALUE **vals,
LEX_STRING **names);
CASSANDRA_TYPE_DEF * get_cassandra_field_def(char *cass_name,
int cass_name_length);
public:
int open(const char *name, int mode, uint test_if_locked);
int close(void);

int write_row(uchar *buf);
int update_row(const uchar *old_data, uchar *new_data);
int delete_row(const uchar *buf);

/** @brief
Unlike index_init(), rnd_init() can be called two consecutive times
without rnd_end() in between (it only makes sense if scan=1). In this
case, the second call should prepare for the new table scan (e.g if
rnd_init() allocates the cursor, the second call should position the
cursor to the start of the table; no need to deallocate and allocate
it again. This is a required method.
*/
int rnd_init(bool scan); //required
int rnd_end();
int rnd_next(uchar *buf); ///< required
int rnd_pos(uchar *buf, uchar *pos); ///< required
void position(const uchar *record); ///< required
int info(uint); ///< required
int delete_all_rows(void);
ha_rows records_in_range(uint inx, key_range *min_key,
key_range *max_key);
int create(const char *name, TABLE *form,
HA_CREATE_INFO *create_info); ///< required
bool check_if_incompatible_data(HA_CREATE_INFO *info,
uint table_changes);

THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to,
enum thr_lock_type lock_type); ///< required

my_bool register_query_cache_table(THD *thd, char *table_key,
uint key_length,
qc_engine_callback
*engine_callback,
ulonglong *engine_data)
{
/*
Do not put data from Cassandra tables into query cache (because there
is no way to tell whether the data in cassandra cluster has changed or
not)
*/
return FALSE;
}
};

 

I will next study the Datastax C++ driver and do some practice with it to see how it communicates with Cassandra.

 

GSoC 2016 Cassandra Storage Engine V2

This summer, I will be working with the MariaDB project to upgrade an existing storage engine to use a new Library released by Datastax. I will be mentored by Sergey Petrunia.

I will be working on the Cassandra storage engine which is used as an interface between MariaDB and Cassandra clusters.  MariaDB has a pluggable architecture which allows the development of new storage engines that can be loaded at runtime without modifying the server code. This is achieved by implementing an instance of the handler interface.

I this project, I will base my work on MariaDB 10.1 which I build from source using steps outlined in  KB.