> ## Documentation Index
> Fetch the complete documentation index at: https://liquidai-alay2shah-sync-notebook-snippets.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Messages & Content

> API reference for chat messages and content types in the LEAP Android SDK

## `ChatMessage`

Data class that is compatible with the message object in OpenAI chat completion API.

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
data class ChatMessage(
  val role: Role,
  val content: List<ChatMessageContent>
  val reasoningContent: String? = null
  val functionCalls: List<LeapFunctionCall>? = null,
) {
  fun toJSONObject(): JSONObject
}

ChatMessage.fromJSONObject(obj: JSONObject): ChatMessage
```

### Fields

* `role`: The role of this message (see [`ChatMessage.Role`](#chatmessagerole)).
* `content`: A list of message contents. Each element is an instance of [`ChatMessageContent`](#chatmessagecontent).
* `reasoningContent`: The reasoning content generated by the reasoning models. Only messages generated by reasoning models will have this field. For other models or other roles, this field should be `null`.
* `functionCalls`: Function call requests generated by the model. See [Function Calling](./function-calling) guide for more details.

### `toJSONObject`

Return a `JSONObject` that represents the chat message. The returned object is compatible with `ChatCompletionRequestMessage` from OpenAI API. It contains 2 fields: `role` and `content` .

See also: [Serialization Support](./utilities#serialization-support).

### `fromJSONObject`

Construct a `ChatMessage` instance from a `JSONObject`. Not all JSON object variants in `ChatCompletionRequestMessage` of OpenAI API are acceptable. As of now, `role` supports `user`, `system` and `assistant`; `content` can be a string or an array.

<Info>
  `LeapSerializationException` will be thrown if the provided JSONObject cannot be recognized as a
  message.
</Info>

See also: [Serialization Support](./utilities#serialization-support).

### `ChatMessage.Role`

Roles of the chat messages, which follows the OpenAI API definition. It is an enum with the following values:

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
enum class Role(val type: String) {
  SYSTEM("system"),
  USER("user"),
  ASSISTANT("assistant"),
}
```

* `SYSTEM`: Indicates the associated content is part of the system prompt. It is generally the first message, to provide guidance on how the model should behave.
* `USER`: Indicates the associated content is user input.
* `ASSISTANT`: Indicates the associated content is model-generated output.

## `ChatMessageContent`

Data class that is compatible with the content object in OpenAI chat completion API. It is a sealed interface.

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
abstract interface ChatMessageContent {
  fun clone(): ChatMessageContent
  fun toJSONObject(): JSONObject
}
fun ChatMessageContent.fromJSONObject(obj: JSONObject): ChatMessageContent

data class ChatMessageContent.Text(val text: String): ChatMessageContent
data class ChatMessageContent.Image(val jpegByteArray: ByteArray): ChatMessageContent
data class ChatMessageContent.Audio(val wavByteArray: ByteArray): ChatMessageContent
```

* `toJSONObject` returns an OpenAI API compatible content object (with a `type` field and the real content fields)
* `fromJSONObject` receives an OpenAI API compatible content object to build a message content. Not all OpenAI content objects are accepted.

Currently, only following content types are supported:

* `Text` : pure text content.
* `Image`: JPEG-encoded image content.
* `Audio`: WAV-encoded audio content

<Info>
  `LeapSerializationException` will be thrown if the provided JSONObject cannot be recognized as a
  message.
</Info>

### `ChatMessageContent.Text`

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
data class ChatMessageContent.Text(val text: String): ChatMessageContent
```

Pure text content. The content is available in the `text` field.

### `ChatMessageContent.Image`

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
data class ChatMessageContent.Image(val jpegByteArray: ByteArray): ChatMessageContent {
  companion object {
    suspend fun fromBitmap(
      bitmap: android.graphics.Bitmap,
      compressionQuality: Int = 85,
    ): ChatMessageContent.Image
  }
}
```

Image content. Currently, we only support JPEG encoded data. `fromBitmap` helper function can create an `ChatMessageContent.Image` content from an Android `Bitmap` object, but the image will be compressed.

<Warning>
  Only the models with vision capabilities can process image content. Sending image content to other
  models may result in unexpected outputs or errors.
</Warning>

### `ChatMessageContent.Audio`

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
data class Audio(val wavByteArray: ByteArray) : ChatMessageContent {
  constructor(data: ByteArray) : this(inputAudio = InputAudio(data = data))
}
```

Audio content for speech recognition and audio understanding. The inference engine requires **WAV-encoded audio** with specific format requirements.

#### Audio Format Requirements

The LEAP inference engine expects WAV files with the following specifications:

| Property        | Required Value                | Notes                                                    |
| --------------- | ----------------------------- | -------------------------------------------------------- |
| **Format**      | WAV (RIFF)                    | Only WAV format is supported                             |
| **Sample Rate** | 16000 Hz (16 kHz) recommended | Other sample rates are automatically resampled to 16 kHz |
| **Encoding**    | PCM (various bit depths)      | Supports Float32, Int16, Int24, Int32                    |
| **Channels**    | Mono (1 channel)              | **Required** - stereo audio will be rejected             |
| **Byte Order**  | Little-endian                 | Standard WAV format                                      |

**Supported PCM Encodings:**

* **Float32**: 32-bit floating point, normalized to \[-1.0, 1.0]
* **Int16**: 16-bit signed integer, range \[-32768, 32767] (recommended)
* **Int24**: 24-bit signed integer, range \[-8388608, 8388607]
* **Int32**: 32-bit signed integer, range \[-2147483648, 2147483647]

<Warning>
  The inference engine **only accepts WAV format**. MP3, AAC, OGG, or other compressed formats are not supported and will cause errors. Audio must be converted to WAV before sending to the model.
</Warning>

<Info>
  **Automatic Resampling**: The inference engine automatically resamples audio to 16 kHz if provided at a different sample rate. However, for best performance and quality, provide audio at 16 kHz to avoid resampling overhead.
</Info>

<Warning>
  **Mono Channel Required**: The inference engine strictly requires single-channel (mono) audio. Multi-channel or stereo WAV files will be rejected with an error. Convert stereo audio to mono before sending.
</Warning>

#### Creating Audio Content from WAV Files

**From a WAV file:**

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
val audioFile = File("/path/to/audio.wav")
val wavBytes = audioFile.readBytes()
val audioContent = ChatMessageContent.Audio(wavBytes)

val message = ChatMessage(
    role = ChatMessage.Role.USER,
    content = listOf(
        ChatMessageContent.Text("What is being said in this audio?"),
        audioContent
    )
)
```

#### Creating Audio Content from Raw PCM Samples

If you're recording audio or have raw PCM data, use the `FloatAudioBuffer` utility to create properly formatted WAV files:

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
import ai.liquid.leap.audio.FloatAudioBuffer

// Collect audio samples (32-bit float PCM, normalized to -1.0 to 1.0)
val audioBuffer = FloatAudioBuffer(sampleRate = 16000)

// Add audio chunks as they arrive
audioBuffer.add(floatArrayOf(0.1f, 0.2f, 0.15f, ...))
audioBuffer.add(floatArrayOf(0.3f, 0.25f, ...))

// Create WAV-encoded bytes
val wavBytes = audioBuffer.createWavBytes()
val audioContent = ChatMessageContent.Audio(wavBytes)
```

<Info>
  `FloatAudioBuffer` automatically creates a valid WAV header and encodes the samples as 32-bit float PCM in a WAV container, which is compatible with the inference engine.
</Info>

#### Recording Audio on Android

When recording audio from the device microphone, configure `AudioRecord` or use a library like `WaveRecorder` with the correct settings:

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
import com.github.squti.androidwaverecorder.WaveRecorder

val waveRecorder = WaveRecorder(outputFilePath)
waveRecorder.configureWaveSettings {
    sampleRate = 16000                                      // 16 kHz
    channels = android.media.AudioFormat.CHANNEL_IN_MONO    // Mono
    audioEncoding = android.media.AudioFormat.ENCODING_PCM_16BIT  // 16-bit PCM
}

waveRecorder.startRecording()
// ... wait for user to finish speaking ...
waveRecorder.stopRecording()

// Read the WAV file
val audioFile = File(outputFilePath)
val wavBytes = audioFile.readBytes()
val audioContent = ChatMessageContent.Audio(wavBytes)
```

#### Audio Duration Considerations

* **Minimum duration**: At least 1 second of audio is recommended for reliable speech recognition
* **Maximum duration**: Limited by the model's context window (typically several minutes)
* **Silence**: Trim excessive silence from the beginning and end for better results

#### Audio Output from Models

When generating audio responses (e.g., with `LFM2.5-Audio-1.5B`), the model outputs audio at **24 kHz sample rate**:

```kotlin theme={"theme":{"light":"github-light","dark":"github-dark"}}
conversation.generateResponse(userMessage)
    .onEach { response ->
        when (response) {
            is MessageResponse.AudioSample -> {
                // samples: FloatArray (32-bit float PCM)
                // sampleRate: Int (typically 24000 Hz for audio generation models)
                val samples = response.samples
                val sampleRate = response.sampleRate

                // Accumulate or play audio samples
                audioBuffer.add(samples)
            }
        }
    }
    .collect()
```

<Info>
  **Note**: Audio **input** should be 16 kHz, but audio **output** from generation models is typically 24 kHz. Make sure your audio playback code supports the correct sample rate.
</Info>